Add, remove, or reconfigure Ceph OSDs¶
Mirantis Ceph Controller simplifies Ceph cluster management by automating LCM operations. This section describes how to add, remove, or reconfigure Ceph OSDs.
Add a Ceph OSD on a managed cluster¶
Manually prepare the required machine devices with LVM2 on the existing node because
BareMetalHostProfile
does not support in-place changes.Open the
KaasCephCluster
CR of a managed cluster for editing:kubectl edit kaascephcluster -n <managedClusterProjectName>
Substitute
<managedClusterProjectName>
with the corresponding value.In the
nodes.<machineName>.storageDevices
section, specify the parameters for a Ceph OSD as required. For the parameters description, see Node parameters.For example:
nodes: kaas-mgmt-node-5bgk6: ... storageDevices: - config: deviceClass: hdd name: sdb
Verify that the Ceph OSD on the specified node is successfully deployed. The
fullClusterInfo
section should not contain any issues.kubectl -n <managedClusterProjectName> get kaascephcluster -o yaml
For example:
status: fullClusterInfo: cephDetails: cephDeviceMapping: kaas-node-d4aac64d-1721-446c-b7df-e351c3025591: "10": "sdb" daemonsStatus: ... osd: running: '3/3 running: 3 up, 3 in' status: Ok
Verify the Ceph OSD on the managed cluster:
kubectl -n rook-ceph get pod -l app=rook-ceph-osd -o wide | grep <machineName>
Remove a Ceph OSD from a managed cluster¶
Note
Ceph OSD removal presupposes usage of a KaaSCephOperationRequest
CR. For workflow overview, spec and phases description, see
High-level workflow of Ceph OSD or node removal.
Warning
When using the non-recommended Ceph pools replicated.size
of
less than 3
, Ceph OSD removal cannot be performed. The minimal replica
size equals a rounded up half of the specified replicated.size
. For
example, if replicated.size
is 2
, the minimal replica size is 1
,
and if replicated.size
is 3
, then the minimal replica size is 2
.
The replica size of 1
allows Ceph having PGs with only one Ceph OSD in
the acting
state, which may cause a PG_TOO_DEGRADED
health warning
that blocks Ceph OSD removal.
Mirantis recommends having setting replicated.size
to 3
for each
Ceph pool.
Open the
KaasCephCluster
CR of a managed cluster for editing:kubectl edit kaascephcluster -n <managedClusterProjectName>
Substitute
<managedClusterProjectName>
with the corresponding value.Remove the required Ceph OSD specification from the
spec.cephClusterSpec.nodes.<machineName>.storageDevices
list:For example:
spec: cephClusterSpec: nodes: <machineName>: ... storageDevices: - config: # remove the entire item entry from storageDevices list deviceClass: hdd name: sdb
Create a YAML template for the
KaaSCephOperationRequest
CR. For example:apiVersion: kaas.mirantis.com/v1alpha1 kind: KaaSCephOperationRequest metadata: name: remove-osd-<machineName>-sdb namespace: <managedClusterProjectName> spec: osdRemove: nodes: <machineName>: cleanupByDevice: - name: sdb kaasCephCluster: name: <kaasCephClusterName> namespace: <managedClusterProjectName>
Substitute
<managedClusterProjectName>
with the corresponding cluster namespace and<kaasCephClusterName>
with the correspondingKaaSCephCluster
name.Note
Since Container Cloud 2.23.0 and 2.23.1 for MOSK 23.1,
cleanupByDevice
is not supported if a device was physically removed from a node. Therefore, usecleanupByOsdId
instead. For details, see Remove a failed Ceph OSD by Ceph OSD ID.Since Container Cloud 2.19.0 and 2.20.1 for MOSK 22.4, if the
storageDevice
item was specified withby-id
, specify thepath
parameter in thecleanupByDevice
section instead ofname
.If the
storageDevice
item was specified with aby-path
device path, specify thepath
parameter in thecleanupByDevice
section instead ofname
.
Apply the template on the management cluster in the corresponding namespace:
kubectl apply -f remove-osd-<machineName>-sdb.yaml
Verify that the corresponding request has been created:
kubectl get kaascephoperationrequest remove-osd-<machineName>-sdb -n <managedClusterProjectName>
Verify that the
removeInfo
section appeared in theKaaSCephOperationRequest
CRstatus
:kubectl -n <managedClusterProjectName> get kaascephoperationrequest remove-osd-<machineName>-sdb -o yaml
Example of system response:
status: childNodesMapping: kaas-node-d4aac64d-1721-446c-b7df-e351c3025591: <machineName> osdRemoveStatus: removeInfo: cleanUpMap: kaas-node-d4aac64d-1721-446c-b7df-e351c3025591: osdMapping: "10": deviceMapping: sdb: path: "/dev/disk/by-path/pci-0000:00:1t.9" partition: "/dev/ceph-b-vg_sdb/osd-block-b-lv_sdb" type: "block" class: "hdd" zapDisk: true
Verify that the
cleanUpMap
section matches the required removal and wait for theApproveWaiting
phase to appear instatus
:kubectl -n <managedClusterProjectName> get kaascephoperationrequest remove-osd-<machineName>-sdb -o yaml
Example of system response:
status: phase: ApproveWaiting
Edit the
KaaSCephOperationRequest
CR and set theapprove
flag totrue
:kubectl -n <managedClusterProjectName> edit kaascephoperationrequest remove-osd-<machineName>-sdb
For example:
spec: osdRemove: approve: true
Review the status of the
KaaSCephOperationRequest
resource request processing. The valuable parameters are as follows:status.phase
- the current state of request processingstatus.messages
- the description of the current phasestatus.conditions
- full history of request processing before the current phasestatus.removeInfo.issues
andstatus.removeInfo.warnings
- contain error and warning messages occurred during request processing
Verify that the
KaaSCephOperationRequest
has been completed. For example:status: phase: Completed # or CompletedWithWarnings if there are non-critical issues
Remove the device cleanup jobs:
kubectl delete jobs -n ceph-lcm-mirantis -l app=miraceph-cleanup-disks
Reconfigure a Ceph OSD on a managed cluster¶
There is no hot reconfiguration procedure for existing Ceph OSDs. To reconfigure an existing Ceph node, follow the steps below:
Remove a Ceph OSD from the Ceph cluster as described in Remove a Ceph OSD from a managed cluster.
Add the same Ceph OSD but with a modified configuration as described in Add a Ceph OSD on a managed cluster.