Add, remove, or reconfigure Ceph OSDs with metadata devices
Pelagia Lifecycle Management (LCM) Controller simplifies Ceph cluster management by automating LCM operations. This section describes how to add, remove, or reconfigure Ceph OSDs with a separate metadata device.
Add a Ceph OSD with a metadata device
Configure one disk for data and one logical volume for metadata of a Ceph OSD to be added to the Ceph cluster.
Note
If you add a new disk after node provisioning, manually prepare the required node devices using Logical Volume Manager (LVM) 2 on the existing node.
Note
You can add a Ceph OSD on top of a raw device that is already present on a node or that you add to the node later. To add a raw device to a node:
Add the device during node deployment.
If the node supports adding devices without a reboot, hot plug the device.
If the node does not support hot plugging, shut down the node, add the device, and start the node again.
Open the
CephDeploymentcustom resource (CR) for editing:kubectl -n ceph-lcm-mirantis edit cephdpl
In the
nodes.<nodeName>.devicessection, specify the parameters for a Ceph OSD as required. For the parameters description, see Nodes parameters.The example configuration of the
nodessection with the new node:nodes: - name: storage-worker-505 roles: - mon - mgr devices: - config: # existing item deviceClass: hdd fullpath: /dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS - config: # new item deviceClass: hdd metadataDevice: /dev/bluedb/meta_1 fullpath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040AL_PN1334PEHN1VBC
Verify that the Ceph OSD is successfully deployed on the specified node. The
CephDeploymentHealthCRstatus.healthReport.cephDaemons.cephDaemonssection should not contain any issues:kubectl -n ceph-lcm-mirantis get cephdeploymenthealth -o yaml
For example:
status: healthReport: cephDaemons: cephDaemons: osd: info: - 4 osds, 4 up, 4 in status: ok
Verify the Ceph OSD status:
kubectl -n rook-ceph get pod -l app=rook-ceph-osd -o wide | grep <nodeName>
Substitute
<nodeName>with the corresponding node name.Example of system response:
rook-ceph-osd-0-7b8d4d58db-f6czn 1/1 Running 0 42h 10.100.91.6 kaas-node-6c5e76f9-c2d2-4b1a-b047-3c299913a4bf <none> <none> rook-ceph-osd-1-78fbc47dc5-px9n2 1/1 Running 0 21h 10.100.91.6 kaas-node-6c5e76f9-c2d2-4b1a-b047-3c299913a4bf <none> <none> rook-ceph-osd-3-647f8d6c69-87gxt 1/1 Running 0 21h 10.100.91.6 kaas-node-6c5e76f9-c2d2-4b1a-b047-3c299913a4bf <none> <none>
Remove a Ceph OSD with a metadata device
Ceph OSD removal presupposes usage of a CephOsdRemoveTask CR. For workflow overview, see Create a Ceph OSD remove task.
Warning
We do not recommend using replicated Ceph pools with replicated.size of less
than 3 in production environments. For such pools, Ceph OSD removal may cause
issues. If replicated.size is 1, Ceph LCM operations cannot function correctly.
Set replicated.size to 3 for each replicated Ceph pool. For erasure-coded pools,
the minimal recommended configuration is 2+1, that is,
erasureCoded.dataChunks: 2 and erasureCoded.codingChunks: 1.
Open the
CephDeploymentobject for editing:kubectl -n ceph-lcm-mirantis edit cephdpl
Remove the required Ceph OSD specification from the
spec.nodes.<nodeName>.deviceslist.Example configuration of the
nodessection with the new node:nodes: - name: storage-worker-505 roles: - mon - mgr devices: - config: deviceClass: hdd fullpath: /dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS - config: # remove the entire item entry from devices list deviceClass: hdd metadataDevice: /dev/bluedb/meta_1 fullpath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040AL_PN1334PEHN1VBC
Create a YAML template for the
CephOsdRemoveTaskCR. Select from the following options:Remove Ceph OSD by device name,
by-pathsymlink, orby-idsymlink:apiVersion: lcm.mirantis.com/v1alpha1 kind: CephOsdRemoveTask metadata: name: remove-osd-<nodeName>-task namespace: ceph-lcm-mirantis spec: nodes: <nodeName>: cleanupByDevice: - device: sdb - device: sdc
Warning
We do not recommend setting device name or device
by-pathsymlink in thecleanupByDevicefield as these identifiers are not persistent and can change at node boot. Remove Ceph OSDs withby-idsymlinks or usecleanupByOsdinstead. For details, see Addressing Ceph storage devices.Note
If a device was physically removed from a node,
cleanupByDeviceis not supported. Therefore, usecleanupByOsdinstead.Remove Ceph OSD by OSD ID:
apiVersion: lcm.mirantis.com/v1alpha1 kind: CephOsdRemoveTask metadata: name: remove-osd-<nodeName>-task namespace: ceph-lcm-mirantis spec: nodes: <nodeName>: cleanupByOsd: - id: 5 - id: 10
Apply the template:
kubectl apply -f remove-osd-<nodeName>-task.yaml
Verify that the corresponding task has been created:
kubectl -n ceph-lcm-mirantis get cephosdremovetask remove-osd-<nodeName>-task
Verify that the
removeInfosection appeared in theCephOsdRemoveTaskCRstatus:kubectl -n ceph-lcm-mirantis get cephosdremovetask remove-osd-<nodeName>-task -o yaml
Example of system response:
status: removeInfo: cleanupMap: storage-worker-505: osdMapping: "10": deviceMapping: sdb: devicePath: "/dev/disk/by-path/pci-0000:00:1t.9" osdPartition: "/dev/ceph-b-vg_sdb/osd-block-b-lv_sdb" osdPartitionType: "block" rotational: true deviceCleanup: true "5": deviceMapping: /dev/sdc: devicePath: /dev/disk/by-path/pci-0000:00:0f.0 osdPartition: /dev/ceph-2d11bf90-e5be-4655-820c-fb4bdf7dda63/osd-block-e41ce9a8-4925-4d52-aae4-e45167cfcf5c osdPartitionType: block rotational: true deviceCleanup: true /dev/sdf: devicePath: /dev/disk/by-path/pci-0000:00:12.0 osdPartition: /dev/bluedb/meta_1 osdPartitionType: db rotational: true
Verify that the
cleanupMapsection matches the required removal and wait for theApproveWaitingphase to appear instatus:kubectl -n ceph-lcm-mirantis get cephosdremovetask remove-osd-<nodeName>-task -o yaml
Example of system response:
status: phase: ApproveWaiting
In the
CephOsdRemoveTaskCR, set theapproveflag totrue:kubectl -n ceph-lcm-mirantis edit cephosdremovetask remove-osd-<nodeName>-task
Configuration snippet:
spec: approve: true
Review the following
statusfields of the Ceph LCM CR processing:status.phase- current state of task processing;status.messages- description of the current phase;status.conditions- full history of task processing before the current phase;status.removeInfo.issuesandstatus.removeInfo.warnings- error and warning messages occurred during task processing, if any.
Verify that the
CephOsdRemoveTaskhas been completed.Example of the positive
status.phasefield:status: phase: Completed # or CompletedWithWarnings if there are non-critical issues
Remove the device cleanup jobs:
kubectl delete jobs -n ceph-lcm-mirantis -l app=pelagia-lcm-cleanup-disks
Reconfigure a partition of a Ceph OSD metadata device
There is no hot reconfiguration procedure for existing Ceph OSDs. To reconfigure an existing Ceph node, remove and re-add a Ceph OSD with a metadata device. However, the automated LCM will clean up the logical volume without a removal, and it can be reused. For this reason, to reconfigure a partition of a Ceph OSD metadata device:
Remove a Ceph OSD from the Ceph cluster.
Add the same Ceph OSD but with a modified configuration.