CephOsdRemoveTask failure with a timeout during rebalance
Ceph OSD removal procedure includes the Ceph OSD out action that starts
the Ceph PGs rebalancing process. The total time for rebalancing depends on a
cluster hardware configuration: network bandwidth, Ceph PGs placement, number
of Ceph OSDs, and so on. The default rebalance timeout is limited by 30
minutes, which applies to standard cluster configurations.
If the rebalance takes more than 30 minutes, the CephOsdRemoveTask
resources created for removing Ceph OSDs or nodes fail with the following
example status:
status:
phase: Failed
messages:
- "cephosdremovetask moved to 'Failed' phase: osd remove failed"
removeInfo:
issues:
- "[node '<node>'] failed to remove osd '2'"
cleanupMap:
<node>:
osdMapping:
"2":
removeStatus:
osdRemoveStatus:
status: Failed
error: timeout (30m0s) reached for waiting pg rebalance
The same timeout also limits the wait for the ceph osd ok-to-stop check
before the Ceph OSD is moved out. In this case, the error field contains
timeout (30m0s) reached for waiting ok-to-stop on osd '2'.
To resolve the issue, increase the timeout for all future
CephOsdRemoveTask resources:
Update pelagia-lcmconfig ConfigMap in Pelagia namespace with the key TASK_OSD_PG_REBALANCE_TIMEOUT_MIN and desired
timeout in minutes in string format:
kubectl -n ceph-lcm-mirantis edit cm pelagia-lcmconfig
Example configuration:
data:
TASK_OSD_PG_REBALANCE_TIMEOUT_MIN: "180"
where "180" means 180 minutes before timeout.
If you have an existing CephOsdRemoveTask resource with issues in
status.removeInfo.issues to process:
In the failed
CephOsdRemoveTaskresource, copy thespecsection.Create a new
CephOsdRemoveTaskwith a different name. For details, see Create a Ceph OSD remove task.Paste the previously copied
specsection of the failedCephOsdRemoveTaskresource to the new one.Remove the failed
CephOsdRemoveTaskresource.