Add, remove, or reconfigure Ceph OSDs

Pelagia Lifecycle Management (LCM) Controller simplifies Ceph cluster management by automating LCM operations. This section describes how to add, remove, or reconfigure Ceph OSDs.

Add a Ceph OSD

  1. Manually prepare the required devices on the existing node.

    Note

    You can add a Ceph OSD on top of a raw device that is already present on a node or that you add to the node later. To add a raw device to a node:

    • Add the device during node deployment.

    • If the node supports adding devices without a reboot, hot plug the device.

    • If the node does not support hot plugging, shut down the node, add the device, and start the node again.

  2. Open the CephDeployment custom resource (CR) for editing:

    kubectl -n ceph-lcm-mirantis edit cephdpl
    
  3. In one of the following sections, specify parameters for Ceph OSD:

    • nodes.<nodeName>.devices

    • nodes.<nodeName>.deviceFilter

    • nodes.<nodeName>.devicePathFilter

    For description of parameters, see Nodes parameters.

    Example configuration of the `nodes` section with the new node:
    ```yaml
    nodes:
    - name: storage-worker-52
      roles:
      - mon
      - mgr
      devices:
      - config: # existing item
          deviceClass: hdd
        fullpath: /dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS
      - config: # new item
          deviceClass: hdd
        fullpath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040AL_PN1334PEHN1VBC
    ```
    
  4. Verify that the Ceph OSD on the specified node is successfully deployed. The CephDeploymentHealth CR status.healthReport.cephDaemons.cephDaemons section should not contain any issues.

    kubectl -n ceph-lcm-mirantis get cephdeploymenthealth -o yaml
    

    For example:

    status:
      healthReport:
        cephDaemons:
          cephDaemons:
            osd:
              info:
              - 3 osds, 3 up, 3 in
              status: ok
    
  5. Verify the desired Ceph OSD pod is Running:

    kubectl -n rook-ceph get pod -l app=rook-ceph-osd -o wide | grep <nodeName>
    

Remove a Ceph OSD

Ceph OSD removal presupposes usage of a CephOsdRemoveTask CR. For workflow overview, see Create a Ceph OSD remove task.

Warning

We do not recommend using replicated Ceph pools with replicated.size of less than 3 in production environments. For such pools, Ceph OSD removal may cause issues. If replicated.size is 1, Ceph LCM operations cannot function correctly. Set replicated.size to 3 for each replicated Ceph pool. For erasure-coded pools, the minimal recommended configuration is 2+1, that is, erasureCoded.dataChunks: 2 and erasureCoded.codingChunks: 1.

  1. Open the CephDeployment CR on for editing:

    kubectl -n ceph-lcm-mirantis edit cephdpl
    
  2. Remove the required Ceph OSD specification from the spec.nodes.<nodeName>.devices list.

    Example configuration of the nodes section with removing device:

    nodes:
    - name: storage-worker-52
      roles:
      - mon
      - mgr
      devices:
      - config:
          deviceClass: hdd
        fullpath: /dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS
      - config: # remove the entire item entry from devices list
          deviceClass: hdd
        fullpath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040AL_PN1334PEHN1VBC
    
  3. Create a YAML template for the CephOsdRemoveTask CR. Select from the following options:

    • Remove Ceph OSD by device name, by-path symlink, or by-id symlink:

      apiVersion: lcm.mirantis.com/v1alpha1
      kind: CephOsdRemoveTask
      metadata:
        name: remove-osd-<nodeName>-task
        namespace: ceph-lcm-mirantis
      spec:
        nodes:
          <nodeName>:
            cleanupByDevice:
            - device: sdb
      

      Warning

      We do not recommend setting device name or device by-path symlink in the cleanupByDevice field as these identifiers are not persistent and can change at node boot. Remove Ceph OSDs with by-id symlinks or use cleanupByOsd instead. For details, see Addressing Ceph storage devices.

      Note

      If a device was physically removed from a node, cleanupByDevice is not supported. Therefore, use cleanupByOsd instead.

    • Remove Ceph OSD by OSD ID:

      apiVersion: lcm.mirantis.com/v1alpha1
      kind: CephOsdRemoveTask
      metadata:
        name: remove-osd-<nodeName>-task
        namespace: ceph-lcm-mirantis
      spec:
        nodes:
          <nodeName>:
            cleanupByOsd:
            - id: 2
      
  4. Apply the template:

    kubectl apply -f remove-osd-<nodeName>-task.yaml
    
  5. Verify that the corresponding request has been created:

    kubectl -n ceph-lcm-mirantis get cephosdremovetask remove-osd-<nodeName>-task
    
  6. Verify that the removeInfo section appeared in the CephOsdRemoveTask CR status:

    kubectl -n ceph-lcm-mirantis get cephosdremovetask remove-osd-<nodeName>-task -o yaml
    

    Example of system response:

    status:
      removeInfo:
        cleanupMap:
          storage-worker-52:
            osdMapping:
              "2":
                deviceMapping:
                  sdb:
                    devicePath: "/dev/disk/by-path/pci-0000:00:1t.9"
                    osdPartition: "/dev/ceph-b-vg_sdb/osd-block-b-lv_sdb"
                    osdPartitionType: "block"
                    rotational: true
                    deviceCleanup: true
    
  7. Verify that the cleanupMap section matches the required removal and wait for the ApproveWaiting phase to appear in status:

    kubectl -n ceph-lcm-mirantis get cephosdremovetask remove-osd-<nodeName>-task -o yaml
    

    Example of system response:

    status:
      phase: ApproveWaiting
    
  8. Edit the CephOsdRemoveTask CR and set the approve flag to true:

    kubectl -n ceph-lcm-mirantis edit cephosdremovetask remove-osd-<nodeName>-task
    

    For example:

    spec:
      approve: true
    
  9. Review the following status fields of the Ceph LCM CR processing:

    • status.phase - current state of request processing;

    • status.messages - description of the current phase;

    • status.conditions - full history of request processing before the current phase;

    • status.removeInfo.issues and status.removeInfo.warnings - error and warning messages occurred during request processing, if any.

  10. Verify that the CephOsdRemoveTask has been completed. For example:

    status:
      phase: Completed # or CompletedWithWarnings if there are non-critical issues
    
  11. Remove the device cleanup jobs:

    kubectl delete jobs -n ceph-lcm-mirantis -l app=pelagia-lcm-cleanup-disks
    

Reconfigure a Ceph OSD

There is no hot reconfiguration procedure for existing Ceph OSDs. To reconfigure an existing Ceph OSD:

  1. Remove a Ceph OSD from the Ceph cluster.

  2. Add the same Ceph OSD but with a modified configuration.