Replace a failed Ceph OSD

After a physical disk replacement, you can use Rook to redeploy a failed Ceph OSD by restarting rook-operator that triggers the reconfiguration of a managed cluster.

To redeploy a failed Ceph OSD:

  1. Log in to a local machine running Ubuntu 18.04 where kubectl is installed.

  2. Obtain and export kubeconfig of a managed cluster as described in Connect to a Mirantis Container Cloud cluster.

  3. Identify the failed Ceph OSD ID:

    ceph osd tree

  4. Remove the Ceph OSD deployment from the managed cluster:

    kubectl delete deployment -n rook-ceph rook-ceph-osd-<ID>

  5. Connect to the terminal of the ceph-tools pod:

    kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod \
-l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

  6. Remove the failed Ceph OSD from the Ceph cluster:

    ceph osd purge osd.<ID>

  7. Delete the authorization key of the failed Ceph OSD from the Ceph cluster:

    ceph auth del osd.<ID>

  8. Exit from the ceph-tools pod.

  9. Replace the failed disk.

  10. Restart the Rook operator:

    kubectl delete pod $(kubectl -n rook-ceph get pod -l "app=rook-ceph-operator" \
-o jsonpath='{.items[0].metadata.name}') -n rook-ceph

  11. Once rook-operator restarts, verify the Ceph OSD pod state and select from the following options:

    • If the Ceph OSD pod is in the Running state, the Ceph OSD has been successfully replaced. Skip the steps below.

    • If the Ceph OSD pod is still in the CrashLoopBackOff state, switch it to the Running state:

      1. Once CrashLoopBackOff of the affected Ceph OSD pod occurs again, determine which node hosts this pod:

        kubectl -n rook-ceph get pod <failedCephOsdPod> -o jsonpath='{.spec.nodeName}'

        Substitute <failedCephOsdPod> with the name of the failed Ceph OSD pod.

      2. Determine the block device folder used by this Ceph OSD pod and save it as <CephOsdBlockDevice>:

        kubectl -n rook-ceph get pod <failedCephOsdPod> -o jsonpath='{.spec.volumes[?(@.name == "activate-osd")].hostPath.path}'

        Substitute <failedCephOsdPod> with the name of the failed Ceph OSD pod.

      3. SSH to the affected node and obtain the Ceph OSD authorization keyring:

        sudo su
cat <CephOsdBlockDevice>/keyring

        Obtain the key value and save it as as <CephOsdKey>.

      4. Return to the cluster and enter the ceph-tools pod:

        kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

      5. Create a Ceph OSD keyring:

        vi key
# inside editor:
[osd.<ID>]
key = <CephOsdKey>
caps mgr = "allow profile osd"
caps mon = "allow profile osd"
caps osd = "allow *"
# save and close the file

      6. Import the created Ceph OSD authorization key:

        ceph auth import -i key

      7. Verify that the key has been successfully imported:

        ceph auth ls

      8. Exit from the ceph-tools pod.

      9. Delete the CrashLoopBackOff Ceph OSD pod and wait until it is Running:

        kubectl -n rook-ceph delete pod <failedCephOsdPod>
kubectl -n rook-ceph get pod -l app=rook-ceph-osd -w