Enable Multisite for Ceph RGW Object Storage¶
Caution
This feature is available as Technology Preview. Use such configuration for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.
The Ceph Multisite feature allows object storage to replicate its data over multiple Ceph clusters. Using Multisite, such object storage is independent and isolated from another object storage in the cluster. For more details, see Ceph documentation: Multisite.
Warning
Rook does not handle Multisite configuration changes and removal.
Therefore, once you enable Multisite for Ceph RGW Object Storage, perform
such operations manually through the ceph-tools
pod. For details, see
Rook documentation: Multisite cleanup.
To enable the Multisite RGW Object Storage:
Select from the following options:
If you do not have a Container cloud cluster yet, open
kaascephcluster.yaml.template
for editing.If the Container cloud cluster is already deployed, open the
KaasCephCluster
CR of a managed cluster for editing:kubectl edit kaascephcluster -n <managedClusterProjectName>
Substitute
<managedClusterProjectName>
with a corresponding value.
Using the following table, update the
cephClusterSpec.objectStorage.multisite
section specification as required:Multisite parameters¶ Parameter
Description
realms
Technical PreviewList of realms to use, represents the realm namespaces. Includes the following parameters:
name
- the realm name.pullEndpoint
- optional, required only when the master zone is in a different storage cluster. The endpoint, access key, and system key of the system user from the realm to pull from. Includes the following parameters:endpoint
- the endpoint of the master zone in the master zone group.accessKey
- the access key of the system user from the realm to pull from.secretKey
- the system key of the system user from the realm to pull from.
zoneGroups
Technical PreviewThe list of zone groups for realms. Includes the following parameters:
name
- the zone group name.realmName
- the realm namespace name to which the zone group belongs to.
zones
Technical PreviewThe list of zones used within one zone group. Includes the following parameters:
name
- the zone name.metadataPool
- the settings used to create the Object Storage metadata pools. Must use replication. For details, see Pool parameters.dataPool
- the settings to create the Object Storage data pool. Can use replication or erasure coding. For details, see Pool parameters.zoneGroupName
- the zone group name.
For example:
objectStorage: multiSite: realms: - name: realm_from_cluster zoneGroups: - name: zonegroup_from_cluster realmName: realm_from_cluster zones: - name: secondary-zone zoneGroupName: zonegroup_from_cluster metadataPool: failureDomain: host replicated: size: 3 dataPool: erasureCoded: codingChunks: 1 dataChunks: 2 failureDomain: host
Select from the following options:
If you do not need to replicate data from a different storage cluster, do not specify the
pullEndpoint
parameter. The current zone used in theObjectStorage
RGW inKaaSCephCluster
will be the master zone.If a different storage cluster exists and its object storage data must be replicated, specify the same realm and zone group names and the
pullEndpoint
parameter. Additionally, specify the endpoint, access key, and system keys of the system user of the realm from which you need to replicate data. For details, see the step 2.To obtain the endpoint of the cluster zone that must be replicated, run the following command specifying the realm and zone group names of the required master zone:
radosgw-admin zonegroup get --rgw-realm=<USER_NAME> --rgw-zonegroup=<ZONE_GROUP_NAME>
To obtain the access key and the secret key of the system user, run the following command on the required Ceph cluster:
radosgw-admin user info --uid="<USER_NAME>"
For example:
objectStorage: multiSite: realms: - name: realm_from_cluster pullEndpoint: endpoint: http://10.11.0.75:8080 accessKey: DRND5J2SVC9O6FQGEJJF secretKey: qpjIjY4lRFOWh5IAnbrgL5O6RTA1rigvmsqRGSJk zoneGroups: - name: zonegroup_from_cluster realmName: realm_from_cluster zones: - name: secondary-zone zoneGroupName: zonegroup_from_cluster metadataPool: failureDomain: host replicated: size: 3 dataPool: erasureCoded: codingChunks: 1 dataChunks: 2 failureDomain: host
Configure the
zone
RADOS Gateway parameter as described in Enable Ceph RGW Object Storage. LeavedataPool
andmetadataPool
empty. These parameters will be ignored because thezone
block in the Multisite configuration specifies the pools parameters.Note
If Ceph RGW Object Storage in your cluster is not set up for Multisite, see Ceph documentation: Migrating a single site system to multi-site.
For example:
rgw: dataPool: {} gateway: allNodes: false instances: 2 port: 80 securePort: 8443 healthCheck: bucket: disabled: true metadataPool: {} name: store-test-pull preservePoolsOnDelete: false zone: name: "secondary-zone"
Once done, ceph-operator
will create the required resources and Rook will
handle the Multisite configuration. For details, see: Rook documentation:
Object Multisite.