Move a Ceph Monitor daemon to another node¶

Warning

This procedure is valid for MOSK clusters that use the MiraCeph custom resource (CR), which is available since MOSK 25.2 to replace the unsupported KaaSCephCluster resource. And MiraCeph will be automatically migrated to CephDeployment in MOSK 26.1. For details, see Deprecation Notes: KaaSCephCluster API on management clusters.

For the equivalent procedure with the unsupported KaaSCephCluster CR, refer to the following section:

Move a Ceph Monitor daemon to another node

This document describes how to migrate a Ceph Monitor daemon from one node to another without changing the general number of Ceph Monitors in the cluster. In the Ceph Controller concept, migration of a Ceph Monitor means manually removing it from one node and adding it to another.

Consider the following exemplary placement scheme of Ceph Monitors in the nodes spec of the MiraCeph CR:

nodes:
  node-1:
    roles:
    - mon
    - mgr
  node-2:
    roles:
    - mgr

Using the example above, if you want to move the Ceph Monitor from node-1 to node-2 without changing the number of Ceph Monitors, the roles table of the nodes spec must result as follows:

nodes:
  node-1:
    roles:
    - mgr
  node-2:
    roles:
    - mgr
    - mon

However, due to the Rook limitation related to Kubernetes architecture, once you move the Ceph Monitor through the MiraCeph CR, changes will not apply automatically. This is caused by the following Rook behavior:

Rook creates Ceph Monitor resources as deployments with nodeSelector, which binds Ceph Monitor pods to a requested node.
Rook does not recreate new Ceph Monitors with the new node placement if the current mon quorum works.

Therefore, to move a Ceph Monitor to another node, you must also manually apply the new Ceph Monitors placement to the Ceph cluster as described below.

To move a Ceph Monitor to another node:

Open the MiraCeph CR on a MOSK cluster:

kubectl -n ceph-lcm-mirantis edit miraceph

In the nodes spec of the MiraCeph CR, change the mon roles placement without changing the total number of mon roles. For details, see the example above. Note the nodes on which the mon roles have been removed and save the name value of that nodes.
If you perform a MOSK cluster update, follow additional steps:
1. Verify that the following conditions are met before proceeding to the next step:
  - There are at least 2 running and available Ceph Monitors so that the Ceph cluster is accessible during the Ceph Monitor migration:
    kubectl -n rook-ceph get pod -l app=rook-ceph-mon kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph -s
  - The MiraCeph object on the MOSK cluster has the required node with the mon role added in the nodes section of spec:
    kubectl -n ceph-lcm-mirantis get miraceph -o yaml
  - The Ceph NodeWorkloadLock for the required node is created:
    kubectl --kubeconfig child-kubeconfig get nodeworkloadlock -o jsonpath='{range .items[?(@.spec.nodeName == "<desiredNodeName>")]}{@.metadata.name}{"\n"}{end}' | grep ceph
2. Scale the ceph-maintenance-controller deployment to 0 replicas:
```
kubectl -n ceph-lcm-mirantis scale deploy ceph-maintenance-controller --replicas 0
```
3. Manually edit the MOSK cluster node labels: remove the ceph_role_mon label from the obsolete node and add this label to the new node:
```
kubectl label node <obsoleteNodeName> ceph_role_mon-
kubectl label node <newNodeName> ceph_role_mon=true
```
4. Verify that the rook-ceph-operator deployment is scaled to 0 replica:
```
kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas 0
```
Obtain the rook-ceph-mon deployment name placed on the obsolete node using the previously obtained node name:
```
kubectl -n rook-ceph get deploy -l app=rook-ceph-mon -o jsonpath="{.items[?(@.spec.template.spec.nodeSelector['kubernetes\.io/hostname'] == '<nodeName>')].metadata.name}"
```
Substitute <nodeName> with the name of the node where you removed the mon role.

Back up the rook-ceph-mon deployment placed on the obsolete node:

kubectl -n rook-ceph get deploy <rook-ceph-mon-name> -o yaml > <rook-ceph-mon-name>-backup.yaml

Remove the rook-ceph-mon deployment placed on the obsolete node:

kubectl -n rook-ceph delete deploy <rook-ceph-mon-name>

If you perform a MOSK cluster update, follow additional steps:

Enter the ceph-tools pod:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Remove the Ceph Monitor from the Ceph monmap by letter:
```
ceph mon rm <monLetter>
```
Substitute <monLetter> with the old Ceph Monitor letter. For example, mon-b has the letter b.
Verify that the Ceph cluster does not have any information about the the removed Ceph Monitor:
```
ceph mon dump
ceph -s
```
Exit the ceph-tools pod.

Scale up the rook-ceph-operator deployment to 1 replica:

kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas 1

Wait for the missing Ceph Monitor failover process to start:

kubectl -n rook-ceph logs -l app=rook-ceph-operator -f

Example of log extract:

2024-03-01 12:33:08.741215 W | op-mon: mon b NOT found in ceph mon map, failover
2024-03-01 12:33:08.741244 I | op-mon: marking mon "b" out of quorum
...
2024-03-01 12:33:08.766822 I | op-mon: Failing over monitor "b"
2024-03-01 12:33:08.766881 I | op-mon: starting new mon...

Select one of the following options:
If you do not perform a MOSK cluster update
Wait approximately 10 minutes until rook-ceph-operator performs a failover of the Pending mon pod. Inspect the logs during the failover process:
kubectl -n rook-ceph logs -l app=rook-ceph-operator -f
Example of log extract:
2021-03-15 17:48:23.471978 W | op-mon: mon "a" not found in quorum, waiting for timeout (554 seconds left) before failover
Note

If the failover process fails:
1. Scale down the rook-ceph-operator deployment to 0 replicas.
2. Apply the backed-up rook-ceph-mon deployment.
3. Scale back the rook-ceph-operator deployment to 1 replica.
If you perform a MOSK cluster update
1. Scale the rook-ceph-operator deployment to 0 replicas:
  kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas 0
2. Scale the ceph-maintenance-controller deployment to 3 replicas:
  kubectl -n ceph-lcm-mirantis scale deploy ceph-maintenance-controller --replicas 3

Once done, Rook removes the obsolete Ceph Monitor from the node and creates a new one on the specified node with a new letter. For example, if the a, b, and c Ceph Monitors were in quorum and mon-c was obsolete, Rook removes mon-c and creates mon-d. In this case, the new quorum includes the a, b, and d Ceph Monitors.

No results

An error occurred

Move a Ceph Monitor daemon to another node¶