Enable Multisite for Ceph RGW Object Storage¶

Caution

This feature is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.

The Ceph Multisite feature allows object storage to replicate its data over multiple Ceph clusters. Using Multisite, such object storage is independent and isolated from another object storage in the cluster. For more details, see Ceph documentation: Multisite.

Warning

Rook does not handle Multisite configuration changes and removal. Therefore, once you enable Multisite for Ceph RGW Object Storage, perform such operations manually through the ceph-tools pod. For details, see Rook documentation: Multisite cleanup.

To enable the Multisite RGW Object Storage:

Select from the following options:
- If you do not have a Container cloud cluster yet, open kaascephcluster.yaml.template for editing.
- If the Container cloud cluster is already deployed, open the KaasCephCluster CR of a managed cluster for editing:
```
kubectl edit kaascephcluster -n <managedClusterProjectName>
```
  Substitute <managedClusterProjectName> with a corresponding value.

Using the following table, update the cephClusterSpec.objectStorage.multisite section specification as required:

Multisite parameters¶
Parameter	Description
`realms` ^{Technical Preview}	List of realms to use, represents the realm namespaces. Includes the following parameters: `name` - the realm name. `pullEndpoint` - optional, required only when the master zone is in a different storage cluster. The endpoint, access key, and system key of the system user from the realm to pull from. Includes the following parameters: `endpoint` - the endpoint of the master zone in the master zone group. `accessKey` - the access key of the system user from the realm to pull from. `secretKey` - the system key of the system user from the realm to pull from.
`zoneGroups` ^{Technical Preview}	The list of zone groups for realms. Includes the following parameters: `name` - the zone group name. `realmName` - the realm namespace name to which the zone group belongs to.
`zones` ^{Technical Preview}	The list of zones used within one zone group. Includes the following parameters: `name` - the zone name. `metadataPool` - the settings used to create the Object Storage metadata pools. Must use replication. For details, see Pool parameters. `dataPool` - the settings to create the Object Storage data pool. Can use replication or erasure coding. For details, see Pool parameters. `zoneGroupName` - the zone group name.

For example:

objectStorage:
  multiSite:
    realms:
    - name: realm_from_cluster
    zoneGroups:
    - name: zonegroup_from_cluster
      realmName: realm_from_cluster
    zones:
    - name: secondary-zone
      zoneGroupName: zonegroup_from_cluster
      metadataPool:
        failureDomain: host
          replicated:
            size: 3
      dataPool:
        erasureCoded:
          codingChunks: 1
          dataChunks: 2
        failureDomain: host

Select from the following options:

If you do not need to replicate data from a different storage cluster, do not specify the pullEndpoint parameter. The current zone used in the ObjectStorage RGW in KaaSCephCluster will be the master zone.

If a different storage cluster exists and its object storage data must be replicated, specify the same realm and zone group names and the pullEndpoint parameter. Additionally, specify the endpoint, access key, and system keys of the system user of the realm from which you need to replicate data. For details, see the step 2.

To obtain the endpoint of the cluster zone that must be replicated, run the following command specifying the realm and zone group names of the required master zone:
```
radosgw-admin zonegroup get --rgw-realm=<USER_NAME> --rgw-zonegroup=<ZONE_GROUP_NAME>
```
To obtain the access key and the secret key of the system user, run the following command on the required Ceph cluster:
```
radosgw-admin user info --uid="<USER_NAME>"
```

For example:

objectStorage:
  multiSite:
    realms:
    - name: realm_from_cluster
      pullEndpoint:
        endpoint: http://10.11.0.75:8080
        accessKey: DRND5J2SVC9O6FQGEJJF
        secretKey: qpjIjY4lRFOWh5IAnbrgL5O6RTA1rigvmsqRGSJk
    zoneGroups:
    - name: zonegroup_from_cluster
      realmName: realm_from_cluster
    zones:
    - name: secondary-zone
      zoneGroupName: zonegroup_from_cluster
      metadataPool:
        failureDomain: host
        replicated:
          size: 3
      dataPool:
        erasureCoded:
          codingChunks: 1
          dataChunks: 2
        failureDomain: host

Configure the zone RADOS Gateway parameter as described in Enable Ceph RGW Object Storage. Leave dataPool and metadataPool empty. These parameters will be ignored because the zone block in the Multisite configuration specifies the pools parameters.

Note

If Ceph RGW Object Storage in your cluster is not set up for Multisite, see Ceph documentation: Migrating a single site system to multi-site.

For example:
```
rgw:
  dataPool: {}
  gateway:
    allNodes: false
    instances: 2
    port: 80
    securePort: 8443
  healthCheck:
    bucket:
      disabled: true
  metadataPool: {}
  name: store-test-pull
  preservePoolsOnDelete: false
  zone:
    name: "secondary-zone"
```

Once done, ceph-operator will create the required resources and Rook will handle the Multisite configuration. For details, see: Rook documentation: Object Multisite.