Enable Ceph RBD mirroring¶

This section describes how to configure and use RADOS Block Device (RBD) mirroring for Ceph pools using the rbdMirror section in the KaaSCephCluster CR. The feature may be useful if, for example, you have two interconnected managed clusters. Once you enable RBD mirroring, the images in the specified pools will be replicated and if a cluster becomes unreachable, the second one will provide users with instant access to all images. For details, see Ceph Documentation: RBD Mirroring.

Note

Ceph Controller only supports bidirectional mirroring.

To enable Ceph RBD monitoring, follow the procedure below and use the following rbdMirror parameters description:

Ceph rbdMirror section parameters¶
Parameter	Description
`daemonsCount`	Count of `rbd-mirror` daemons to spawn. Mirantis recommends using one instance of the `rbd-mirror` daemon.
`peers`	Optional. List of mirroring peers of an external cluster to connect to. Only a single peer is supported. The `peer` section includes the following parameters: `site` - the label of a remote Ceph cluster associated with the token. `token` - the token that will be used by one site (Ceph cluster) to pull images from the other site. To obtain the token, use the rbd mirror pool peer bootstrap create command. `pools` - optional, a list of pool names to mirror.

To enable Ceph RBD mirroring:

In KaaSCephCluster CRs of both Ceph clusters where you want to enable mirroring, specify positive daemonsCount in the spec.cephClusterSpec.rbdMirror section:
```
spec:
  cephClusterSpec:
    rbdMirror:
      daemonsCount: 1
```
On both Ceph clusters where you want to enable mirroring, wait for the Ceph RBD Mirror daemons to start running:
```
kubectl -n rook-ceph get pod -l app=rook-ceph-rbd-mirror
```
In KaaSCephCluster of both Ceph clusters where you want to enable mirroring, specify the spec.cephClusterSpec.pools.mirroring.mode parameter for all pools that must be mirrored.
Mirroring mode recommendations
- Mirantis recommends using the pool mode for mirroring. For the pool mode, explicitly enable journaling for each image.
- To use the image mirroring mode, explicitly enable mirroring as described in the step 8.
```
spec:
  cephClusterSpec:
    pools:
    - name: image-hdd
      ...
      mirroring:
        mode: pool
    - name: volumes-hdd
      ...
      mirroring:
        mode: pool
```
Obtain the name of an external site to mirror with. On pools with mirroring enabled, the name is typically ceph fsid:
```
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l \
"app=rook-ceph-tools" -o name
rbd mirror pool info <mirroringPoolName>
# or
ceph fsid
```
Substitute <mirroringPoolName> with the name of a pool to be mirrored.
On an external site to mirror with, create a new bootstrap peer token. Execute the following command within the ceph-tools pod CLI:
```
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l \
"app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
rbd mirror pool peer bootstrap create <mirroringPoolName> --site-name <siteName>
```
Substitute <mirroringPoolName> with the name of a pool to be mirrored. In <siteName>, assign a label for the external Ceph cluster that will be used along with mirroring.

For details, see Ceph documentation: Bootstrap peers.

In KaaSCephCluster on the cluster that should mirror pools, specify spec.cephClusterSpec.rbdMirror.peers with the obtained peer and pools to mirror:

spec:
  cephClusterSpec:
    rbdMirror:
      ...
      peers:
      - site: <siteName>
        token: <bootstrapPeer>
        pools: [<mirroringPoolName1>, <mirroringPoolName2>, ...]

Substitute <siteName> with the label assigned to the external Ceph cluster, <bootstrapPeer> with the token obtained in the previous step, and <mirroringPoolName> with names of pools that have the mirroring.mode parameter defined.

For example:

spec:
  cephClusterSpec:
    rbdMirror:
      ...
      peers:
      - site: cluster-b
        token: <base64-string>
        pools:
        - images-hdd
        - volumes-hdd
        - special-pool-ssd

Verify that mirroring is enabled and each pool with spec.cephClusterSpec.pools.mirroring.mode defined has an external peer site:
```
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l \
"app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
rbd mirror pool info <mirroringPoolName>
```
Substitute <mirroringPoolName> with the name of a pool with mirroring enabled.
If you have set the image mirroring mode in the pools section, explicitly enable mirroring for each image with rbd within the pool:

Note

Execute the following command within the ceph-tools pod with ceph and rbd CLI.
```
rbd mirror image enable <poolName>/<imageName> <imageMirroringMode>
```
Substitute <poolName> with the name of a pool with the image mirroring mode, <imageName> with the name of an image stored in the specified pool. Substitute <imageMirroringMode> with one of:
- journal - for mirroring to use the RBD journaling image feature to replicate the image contents. If the RBD journaling image feature is not yet enabled on the image, it will be enabled automatically.
- snapshot - for mirroring to use RBD image mirror-snapshots to replicate the image contents. Once enabled, an initial mirror-snapshot will automatically be created. To create additional RBD image mirror-snapshots, use the rbd command.
For details, see Ceph Documentation: Enable image mirroring.