Maintenance stuck on a compact Ceph cluster during OSD rebalancing

Warning

We do not recommend using replicated Ceph pools with replicated.size of less than 3 in production environments. For such pools, Ceph OSD removal may cause issues. If replicated.size is 1, Ceph LCM operations cannot function correctly. Set replicated.size to 3 for each replicated Ceph pool. For erasure-coded pools, the minimal recommended configuration is 2+1, that is, erasureCoded.dataChunks: 2 and erasureCoded.codingChunks: 1.

When disabling or removing a Ceph node during upgrade or maintenance operations such as rolling reboot, Ceph may not complete rebalancing if only two of three OSD nodes remain active. The CephDeployment object can remain in Maintenance, causing the rebalance process to wait indefinitely for Ceph to become ready.

The issue may only affect environments with a small number of Ceph OSD nodes (for example, three), pool replica count set to one less than the number of storage nodes (replicas=storage_nodes_count-1), and failure domain host.

To resolve the issue, run the following command in the pelagia-ceph-toolbox pod for each Ceph OSD of the affected node:

kubectl -n rook-ceph exec -it deploy/pelagia-ceph-toolbox -- ceph osd reweight <osdId> 0