Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Starting with MOSK 25.2, the MOSK documentation set covers all product layers, including MOSK management (formerly MCC). This means everything you need is in one place. The separate MCC documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Enable Ceph RBD mirroring¶

TechPreview

Warning

This procedure is valid for MOSK clusters that use the MiraCeph custom resource (CR), which is available since MOSK 25.2 to replace the deprecated KaaSCephCluster. For the equivalent procedure with the KaaSCephCluster CR, refer to the following section:

Enable Ceph RBD mirroring

This section describes how to configure and use RADOS Block Device (RBD) mirroring for Ceph pools using the rbdMirror section in the MiraCeph CR. The feature may be useful if, for example, you have two interconnected MOSK clusters. Once you enable RBD mirroring, the images in the specified pools will be replicated and if a cluster becomes unreachable, the second one will provide users with instant access to all images. For details, see Ceph Documentation: RBD Mirroring.

Note

Ceph Controller only supports bidirectional mirroring.

To enable Ceph RBD monitoring, follow the procedure below and use the following rbdMirror parameters description:

Ceph rbdMirror section parameters¶
Parameter	Description
`daemonsCount`	Count of `rbd-mirror` daemons to spawn. Mirantis recommends using one instance of the `rbd-mirror` daemon.
`peers`	Optional. List of mirroring peers of an external cluster to connect to. Only a single peer is supported. The `peer` section includes the following parameters: `site` - the label of a remote Ceph cluster associated with the token. `token` - the token that will be used by one site (Ceph cluster) to pull images from the other site. To obtain the token, use the rbd mirror pool peer bootstrap create command. `pools` - optional, a list of pool names to mirror.

To enable Ceph RBD mirroring:

In MiraCeph CRs of both Ceph clusters where you want to enable mirroring, specify positive daemonsCount in the rbdMirror section:
```
rbdMirror:
  daemonsCount: 1
```
On both Ceph clusters where you want to enable mirroring, wait for the Ceph RBD Mirror daemons to start running:
```
kubectl -n rook-ceph get pod -l app=rook-ceph-rbd-mirror
```
In MiraCeph CRs of both Ceph clusters where you want to enable mirroring, specify the pools.mirroring.mode parameter for all pools that must be mirrored.
Mirroring mode recommendations
- Mirantis recommends using the pool mode for mirroring. For the pool mode, explicitly enable journaling for each image.
- To use the image mirroring mode, explicitly enable mirroring as described in the step 8.
```
pools:
- name: image-hdd
  ...
  mirroring:
    mode: pool
- name: volumes-hdd
  ...
  mirroring:
    mode: pool
```

Obtain the name of an external site to mirror with. On pools with mirroring enabled, the name is typically ceph fsid:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- rbd mirror pool info <mirroringPoolName>
# or
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fsid

Substitute <mirroringPoolName> with the name of a pool to be mirrored.

On an external site to mirror with, create a new bootstrap peer token. Execute the following command within the ceph-tools pod CLI:
```
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- rbd mirror pool peer bootstrap create <mirroringPoolName> --site-name <siteName>
```
Substitute <mirroringPoolName> with the name of a pool to be mirrored. In <siteName>, assign a label for the external Ceph cluster that will be used along with mirroring.

For details, see Ceph documentation: Bootstrap peers.
In MiraCeph CR on the cluster that should mirror pools, specify rbdMirror.peers with the obtained peer and pools to mirror:
```
rbdMirror:
  ...
  peers:
  - site: <siteName>
    token: <bootstrapPeer>
    pools: [<mirroringPoolName1>, <mirroringPoolName2>, ...]
```
Substitute <siteName> with the label assigned to the external Ceph cluster, <bootstrapPeer> with the token obtained in the previous step, and <mirroringPoolName> with names of pools that have the mirroring.mode parameter defined.

For example:
```
rbdMirror:
  ...
  peers:
  - site: cluster-b
    token: <base64-string>
    pools:
    - images-hdd
    - volumes-hdd
    - special-pool-ssd
```
Verify that mirroring is enabled and each pool with pools.mirroring.mode defined has an external peer site:
```
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- rbd mirror pool info <mirroringPoolName>
```
Substitute <mirroringPoolName> with the name of a pool with mirroring enabled.
If you have set the image mirroring mode in the pools section, explicitly enable mirroring for each image with rbd within the pool:
```
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- rbd mirror image enable <poolName>/<imageName> <imageMirroringMode>
```
Substitute <poolName> with the name of a pool with the image mirroring mode, <imageName> with the name of an image stored in the specified pool. Substitute <imageMirroringMode> with one of:
- journal - for mirroring to use the RBD journaling image feature to replicate the image contents. If the RBD journaling image feature is not yet enabled on the image, it will be enabled automatically.
- snapshot - for mirroring to use RBD image mirror-snapshots to replicate the image contents. Once enabled, an initial mirror-snapshot will automatically be created. To create additional RBD image mirror-snapshots, use the rbd command.
For details, see Ceph Documentation: Enable image mirroring.