Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Starting with MOSK 25.2, the MOSK documentation set covers all product layers, including MOSK management (formerly MCC). This means everything you need is in one place. The separate MCC documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Add, remove, or reconfigure Ceph OSDs¶

Warning

This procedure is valid for MOSK clusters that use the MiraCeph custom resource (CR), which is available since MOSK 25.2 to replace the deprecated KaaSCephCluster. For the equivalent procedure with the KaaSCephCluster CR, refer to the following section:

Add, remove, or reconfigure Ceph OSDs

Mirantis Ceph Controller simplifies Ceph cluster management by automating LCM operations. This section describes how to add, remove, or reconfigure Ceph OSDs.

Add a Ceph OSD on a managed cluster¶

Manually prepare the required machine devices with LVM2 on the existing node because BareMetalHostProfile does not support in-place changes.
To add a Ceph OSD to an existing or hot-plugged raw device
If you want to add a Ceph OSD on top of a raw device that already exists on a node or is hot-plugged, add the required device using the following guidelines:
- You can add a raw device to a node during node deployment.
- If a node supports adding devices without node reboot, you can hot plug a raw device to a node.
- If a node does not support adding devices without node reboot, you can hot plug a raw device during node shutdown. In this case, complete the following steps:
  1. Enable maintenance mode on the managed cluster.
  2. Turn off the required node.
  3. Attach the required raw device to the node.
  4. Turn on the required node.
  5. Disable maintenance mode on the managed cluster.
Open the MiraCeph CR on a managed cluster for editing:
```
kubectl -n ceph-lcm-mirantis edit miraceph
```
In one of the following sections, specify parameters for Ceph OSD:
- nodes.<nodeName>.devices
- nodes.<nodeName>.deviceFilter
- nodes.<nodeName>.devicePathFilter
For description of parameters, see Node parameters.

The example configuration of the nodes section with the new node:
```
nodes:
- name: kaas-node-5bgk6
  roles:
  - mon
  - mgr
  devices:
  - config: # existing item
      deviceClass: hdd
    fullPath: /dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS
  - config: # new item
      deviceClass: hdd
    fullPath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040AL_PN1334PEHN1VBC
```
Warning

Mirantis highly recommends using the non-wwn by-id symlinks to specify storage devices in the devices list.

For details, see Addressing storage devices since MOSK 25.2.
Verify that the Ceph OSD on the specified node is successfully deployed. The fullClusterInfo section should not contain any issues.
```
kubectl -n ceph-lcm-mirantis get mchealth -o yaml
```
For example:
```
status:
  fullClusterInfo:
    daemonsStatus:
      ...
      osd:
        running: '3/3 running: 3 up, 3 in'
        status: Ok
```
Note

Since MOSK 23.2, cephDeviceMapping is removed because its large size can potentially exceed the Kubernetes 1.5 MB quota.

Verify the Ceph OSD on the managed cluster:

kubectl -n rook-ceph get pod -l app=rook-ceph-osd -o wide | grep <nodeName>

Remove a Ceph OSD from a managed cluster¶

Note

Ceph OSD removal presupposes usage of a CephOsdRemoveRequest CR. For workflow overview, spec and phases description, see High-level workflow of Ceph OSD or node removal.

Warning

When using the non-recommended Ceph pools replicated.size of less than 3, Ceph OSD removal cannot be performed. The minimal replica size equals a rounded up half of the specified replicated.size.

For example, if replicated.size is 2, the minimal replica size is 1, and if replicated.size is 3, then the minimal replica size is 2. The replica size of 1 allows Ceph having PGs with only one Ceph OSD in the acting state, which may cause a PG_TOO_DEGRADED health warning that blocks Ceph OSD removal. Mirantis recommends setting replicated.size to 3 for each Ceph pool.

Open the MiraCeph CR on a managed cluster for editing:
```
kubectl -n ceph-lcm-mirantis edit miraceph
```

Remove the required Ceph OSD specification from the spec.nodes.<machineName>.storageDevices list:

The example configuration of the nodes section with the new node:

nodes:
- name: kaas-node-5bgk6
  roles:
  - mon
  - mgr
  devices:
  - config:
      deviceClass: hdd
    fullPath: /dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS
  - config: # remove the entire item entry from devices list
      deviceClass: hdd
    fullPath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040AL_PN1334PEHN1VBC

Create a YAML template for the CephOsdRemoveRequest CR. Select from the following options:
- Remove Ceph OSD by device name, by-path symlink, or by-id symlink:
```
apiVersion: lcm.mirantis.com/v1alpha1
kind: CephOsdRemoveRequest
metadata:
  name: remove-osd-<nodeName>-sdb
  namespace: ceph-lcm-mirantis
spec:
  nodes:
    <nodeName>:
      cleanupByDevice:
      - name: sdb
```
  Warning
  
  Since MOSK 23.3, Mirantis does not recommend setting device name or device by-path symlink in the cleanupByDevice field as these identifiers are not persistent and can change at node boot. Remove Ceph OSDs with by-id symlinks specified in the path field or use cleanupByOsdId instead. For details, see Addressing storage devices since MOSK 25.2.
  Note
  - If a device was physically removed from a node, cleanupByDevice is not supported. Therefore, use cleanupByOsdId instead. For details, see Remove a failed Ceph OSD by Ceph OSD ID.
  - If the devices item was specified with a by-path device path, specify the path parameter in the cleanupByDevice section instead of name.
- Remove Ceph OSD by OSD ID:
```
apiVersion: lcm.mirantis.com/v1alpha1
kind: CephOsdRemoveRequest
metadata:
  name: remove-osd-<nodeName>-sdb
  namespace: ceph-lcm-mirantis
spec:
  nodes:
    <nodeName>:
      cleanupByOsdId:
      - 2
```

Apply the template on the managed cluster:

kubectl apply -f remove-osd-<nodeName>-sdb.yaml

Verify that the corresponding request has been created:

kubectl -n ceph-lcm-mirantis get cephosdremoverequest remove-osd-<nodeName>-sdb

Verify that the removeInfo section appeared in the CephOsdRemoveRequest CR status:

kubectl -n ceph-lcm-mirantis get cephosdremoverequest remove-osd-<nodeName>-sdb -o yaml

Example of system response:

status:
  removeInfo:
    cleanUpMap:
      kaas-node-d4aac64d-1721-446c-b7df-e351c3025591:
        osdMapping:
          "10":
            deviceMapping:
              sdb:
                path: "/dev/disk/by-path/pci-0000:00:1t.9"
                partition: "/dev/ceph-b-vg_sdb/osd-block-b-lv_sdb"
                type: "block"
                class: "hdd"
                zapDisk: true

Verify that the cleanUpMap section matches the required removal and wait for the ApproveWaiting phase to appear in status:
```
kubectl -n ceph-lcm-mirantis get cephosdremoverequest remove-osd-<nodeName>-sdb -o yaml
```
Example of system response:
```
status:
  phase: ApproveWaiting
```

Edit the CephOsdRemoveRequest CR and set the approve flag to true:

kubectl -n ceph-lcm-mirantis edit cephosdremoverequest remove-osd-<nodeName>-sdb

For example:

spec:
  approve: true

Review the following status fields of the Ceph LCM CR request processing:
- status.phase - current state of request processing
- status.messages - description of the current phase
- status.conditions - full history of request processing before the current phase
- status.removeInfo.issues and status.removeInfo.warnings - error and warning messages occurred during request processing, if any

Verify that the CephOsdRemoveRequest has been completed. For example:

status:
  phase: Completed # or CompletedWithWarnings if there are non-critical issues

Remove the device cleanup jobs:

kubectl delete jobs -n ceph-lcm-mirantis -l app=miraceph-cleanup-disks

Reconfigure a Ceph OSD on a managed cluster¶

There is no hot reconfiguration procedure for existing Ceph OSDs. To reconfigure an existing Ceph node, follow the steps below:

Remove a Ceph OSD from the Ceph cluster as described in Remove a Ceph OSD from a managed cluster.
Add the same Ceph OSD but with a modified configuration as described in Add a Ceph OSD on a managed cluster.