Remove Ceph OSD manually¶

You may need to manually remove a Ceph OSD, for example, in the following cases:

If you have removed a device or node from the KaaSCephCluster spec.cephClusterSpec.nodes or spec.cephClusterSpec.nodeGroups section with manageOsds set to false.
If you do not want to rely on Ceph LCM operations and want to manage the Ceph OSDs life cycle manually.

To safely remove one or multiple Ceph OSDs from a Ceph cluster, perform the following procedure for each Ceph OSD one by one.

Warning

The procedure presupposes the Ceph OSD disk or logical volumes partition cleanup.

To remove a Ceph OSD manually:

Edit the KaaSCephCluster resource on a management cluster:
```
kubectl --kubeconfig <mgmtKubeconfig> -n <managedClusterProjectName> edit kaascephcluster
```
Substitute <mgmtKubeconfig> with the management cluster kubeconfig and <managedClusterProjectName> with the project name of the managed cluster.
In the spec.cephClusterSpec.nodes section, remove the required storageDevices item of the corresponding node spec. If after removal storageDevices becomes empty and the node spec has no roles specified, also remove the node spec.
Obtain kubeconfig of the managed cluster and provide it as an environment variable:
```
export KUBECONFIG=<pathToManagedKubeconfig>
```

Verify that all Ceph OSDs are up and in, the Ceph cluster is healthy, and no rebalance or recovery is in progress:

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l \
app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}') -- ceph -s

Example of system response:

cluster:
  id:     8cff5307-e15e-4f3d-96d5-39d3b90423e4
  health: HEALTH_OK
  ...
  osd: 4 osds: 4 up (since 10h), 4 in (since 10h)

Stop the rook-ceph/rook-ceph-operator deployment to avoid premature reorchestration of the Ceph cluster:
```
kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas 0
```

Enter the ceph-tools pod:

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l \
app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}') bash

Scale the rook-ceph/rook-ceph-osd-<ID> deployment to 0 replicas:

kubectl -n rook-ceph scale deploy rook-ceph-osd-<ID> --replicas 0

Enter the ceph-tools pod:

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l \
app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}') bash

Remove the Ceph OSD from the Ceph cluster:

ceph osd purge <ID> --yes-i-really-mean-it

Delete the Ceph OSD auth entry, if present. Otherwise, skip this step.
```
ceph auth del osd.<ID>
```
If you have removed the last Ceph OSD on the node and want to remove this node from the Ceph cluster, remove the CRUSH map entry:
```
ceph osd crush remove <nodeName>
```
Substitute <nodeName> with the name of the node where the removed Ceph OSD was placed.
Verify that the failure domain within Ceph OSDs has been removed from the CRUSH map:
```
ceph osd tree
```
If you have removed the node, it will be removed from the CRUSH map.
Exit from the ceph-tools pod:
```
exit
```
Clean up the disk used by the removed Ceph OSD. For details, see official Rook documentation.

Warning

If you are using multiple Ceph OSDs per device or metadata device, make sure that you can clean up the entire disk. Otherwise, instead clean up only the logical volume partitions for the volume group by running lvremove <lvpartion_uuid> any Ceph OSD pod that belongs to the same host as the removed Ceph OSD.
Delete the rook-ceph/rook-ceph-osd-<ID> deployment previously scaled to 0 replicas:
```
kubectl -n rook-ceph delete deploy rook-ceph-osd-<ID>
```
Substitute <ID> with the number of the removed Ceph OSD.
Scale the rook-ceph/rook-ceph-operator deployment to 1 replica and wait for the orchestration to complete:
```
kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas 1
kubectl -n rook-ceph get pod -w
```
Once done, Ceph OSD removal is complete.