Enable multisite for Ceph RGW Object Storage¶

Caution

This feature is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.

The Ceph multisite feature allows object storage to replicate its data over multiple Ceph clusters. Using multisite, such object storage is independent and isolated from another object storage in the cluster. Only the multi-zone multisite setup is currently supported. For more details, see Ceph documentation: Multisite.

Enable the multisite RGW Object Storage¶

Select from the following options:
- If you do not have a Container cloud cluster yet, open kaascephcluster.yaml.template for editing.
- If the Container cloud cluster is already deployed, open the KaasCephCluster CR of a managed cluster for editing:
```
kubectl edit kaascephcluster -n <managedClusterProjectName>
```
  Substitute <managedClusterProjectName> with a corresponding value.

Using the following table, update the cephClusterSpec.objectStorage.multisite section specification as required:

Multisite parameters¶
Parameter	Description
`realms` ^{Technical Preview}	List of realms to use, represents the realm namespaces. Includes the following parameters: `name` - the realm name. `pullEndpoint` - optional, required only when the master zone is in a different storage cluster. The endpoint, access key, and system key of the system user from the realm to pull from. Includes the following parameters: `endpoint` - the endpoint of the master zone in the master zone group. `accessKey` - the access key of the system user from the realm to pull from. `secretKey` - the system key of the system user from the realm to pull from.
`zoneGroups` ^{Technical Preview}	The list of zone groups for realms. Includes the following parameters: `name` - the zone group name. `realmName` - the realm namespace name to which the zone group belongs to.
`zones` ^{Technical Preview}	The list of zones used within one zone group. Includes the following parameters: `name` - the zone name. `metadataPool` - the settings used to create the Object Storage metadata pools. Must use replication. For details, see Pool parameters. `dataPool` - the settings to create the Object Storage data pool. Can use replication or erasure coding. For details, see Pool parameters. `zoneGroupName` - the zone group name. `endpointsForZone` - available since Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0). The list of all endpoints in the zone group. If you use ingress proxy for RGW, the list of endpoints must contain that FQDN/IP address to access RGW. By default, if no ingress proxy is used, the list of endpoints is set to the IP address of the RGW external service. Endpoints must follow the HTTP URL format.

Caution

The multisite configuration requires master and secondary zones to be reachable from each other.

Select from the following options:

If you do not need to replicate data from a different storage cluster, and the current cluster represents the master zone, modify the current objectStorage section to use the multisite mode:

Configure the zone RADOS Gateway (RGW) parameter by setting it to the RGW Object Storage name.

Note

Leave dataPool and metadataPool empty. These parameters are ignored because the zone block in the multisite configuration specifies the pools parameters. Other RGW parameters do not require changes.

For example:

objectStorage:
  rgw:
    dataPool: {}
    gateway:
      allNodes: false
      instances: 2
      port: 80
      securePort: 8443
    healthCheck: {}
    metadataPool: {}
    name: openstack-store
    preservePoolsOnDelete: false
    zone:
      name: openstack-store

Create the multiSite section where the names of realm, zone group, and zone must match the current RGW name.

Since Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0), specify the endpointsForZone parameter according to your configuration:

If you use ingress proxy, which is defined in the spec.cephClusterSpec.ingress section, add the FQDN endpoint.
If you do not use any ingress proxy and access the RGW API using the default RGW external service, add the IP address of the external service or leave this parameter empty.

The following example illustrates a complete objectStorage section:

objectStorage:
  multiSite:
    realms:
    - name: openstack-store
    zoneGroups:
    - name: openstack-store
      realmName: openstack-store
    zones:
    - name: openstack-store
      zoneGroupName: openstack-store
      endpointsForZone: http://10.11.0.75:8080
      metadataPool:
        failureDomain: host
          replicated:
            size: 3
      dataPool:
        erasureCoded:
          codingChunks: 1
          dataChunks: 2
        failureDomain: host
  rgw:
    dataPool: {}
    gateway:
      allNodes: false
      instances: 2
      port: 80
      securePort: 8443
    healthCheck: {}
    metadataPool: {}
    name: openstack-store
    preservePoolsOnDelete: false
    zone:
      name: openstack-store

If you use a different storage cluster, and its object storage data must be replicated, specify the realm and zone group names along with the pullEndpoint parameter. Additionally, specify the endpoint, access key, and system keys of the system user of the realm from which you need to replicate data. For details, see the step 2 of this procedure.

To obtain the endpoint of the cluster zone that must be replicated, run the following command by specifying the zone group name of the required master zone on the master zone side:
```
radosgw-admin zonegroup get --rgw-zonegroup=<ZONE_GROUP_NAME> | jq -r '.endpoints'
```
The endpoint is located in the endpoints field.
To obtain the access key and the secret key of the system user, run the following command on the required Ceph cluster:
```
radosgw-admin user list
```
To obtain the system user name, which has your RGW ObjectStorage name as prefix:
```
radosgw-admin user info --uid="<USER_NAME>" | jq -r '.keys'
```

For example:

objectStorage:
  multiSite:
    realms:
    - name: openstack-store
      pullEndpoint:
        endpoint: http://10.11.0.75:8080
        accessKey: DRND5J2SVC9O6FQGEJJF
        secretKey: qpjIjY4lRFOWh5IAnbrgL5O6RTA1rigvmsqRGSJk
    zoneGroups:
    - name: openstack-store
      realmName: openstack-store
    zones:
    - name: openstack-store-backup
      zoneGroupName: openstack-store
      metadataPool:
        failureDomain: host
        replicated:
          size: 3
      dataPool:
        erasureCoded:
          codingChunks: 1
          dataChunks: 2
        failureDomain: host

Note

Mirantis recommends using the same metadataPool and dataPool settings as you use in the master zone.

Configure the zone RGW parameter and leave dataPool and metadataPool empty. These parameters are ignored because the zone section in the multisite configuration specifies the pools parameters.

Also, you can split the RGW daemon on daemons serving clients and daemons running synchronization. To enable this option, specify splitDaemonForMultisiteTrafficSync in the gateway section.

For example:

objectStorage:
  multiSite:
     realms:
     - name: openstack-store
       pullEndpoint:
         endpoint: http://10.11.0.75:8080
         accessKey: DRND5J2SVC9O6FQGEJJF
         secretKey: qpjIjY4lRFOWh5IAnbrgL5O6RTA1rigvmsqRGSJk
     zoneGroups:
     - name: openstack-store
       realmName: openstack-store
     zones:
     - name: openstack-store-backup
       zoneGroupName: openstack-store
       metadataPool:
         failureDomain: host
         replicated:
           size: 3
       dataPool:
         erasureCoded:
           codingChunks: 1
           dataChunks: 2
         failureDomain: host
  rgw:
    dataPool: {}
    gateway:
      allNodes: false
      instances: 2
      splitDaemonForMultisiteTrafficSync: true
      port: 80
      securePort: 8443
    healthCheck: {}
    metadataPool: {}
    name: openstack-store-backup
    preservePoolsOnDelete: false
    zone:
      name: openstack-store-backup

On the ceph-tools pod, verify the multisite status:
```
radosgw-admin sync status
```

Once done, ceph-operator will create the required resources and Rook will handle the multisite configuration. For details, see: Rook documentation: Object Multisite.

Configure and clean up a multisite configuration¶

Warning

Rook does not handle multisite configuration changes and cleanup. Therefore, once you enable multisite for Ceph RGW Object Storage, perform these operations manually in the ceph-tools pod. For details, see Rook documentation: Multisite cleanup.

If automatic update of zone group hostnames is disabled, manually specify all required hostnames and update the zone group. In the ceph-tools pod, run the following script:

/usr/local/bin/zonegroup_hostnames_update.sh --rgw-zonegroup <ZONEGROUP_NAME> --hostnames fqdn1[,fqdn2]

If the multisite setup is completely cleaned up, manually execute the following steps on the ceph-tools pod:

Remove the .rgw.root pool:
```
ceph osd pool rm .rgw.root .rgw.root --yes-i-really-really-mean-it
```
Some other RGW pools may also require a removal after cleanup.

Remove the related RGW crush rules:

ceph osd crush rule ls | grep rgw | xargs -I% ceph osd crush rule rm %