Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Starting with MOSK 25.2, the MOSK documentation set covers all product layers, including MOSK management (formerly MCC). This means everything you need is in one place. The separate MCC documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Migrate Ceph cluster to address storage devices using by-id¶

Warning

This procedure is valid for MOSK clusters that use the MiraCeph custom resource (CR), which is available since MOSK 25.2 to replace the deprecated KaaSCephCluster. For the equivalent procedure with the KaaSCephCluster CR, refer to the following section:

Migrate Ceph cluster to address storage devices using by-id

The by-id identifier is the only persistent device identifier for a Ceph cluster that remains stable after the cluster upgrade or any other maintenance. Therefore, Mirantis recommends using device by-id symlinks rather than device names or by-path symlinks.

MOSK uses the device by-id identifier as the default method of addressing the underlying devices of Ceph OSDs. Thus, you should migrate all existing Ceph clusters, which are still utilizing the device names or device by-path symlinks, to the by-id format.

This section explains how to configure the MiraCeph specification to use the by-id symlinks instead of disk names and by-path identifiers as the default method of addressing storage devices.

Note

Mirantis recommends avoiding the use of wwn symlinks as by-id identifiers due to their lack of persistence expressed in inconsistent discovery during node boot.

Besides migrating to by-id, consider using the fullPath field for the by-id symlinks configuration, instead of the name field in the spec.nodes.devices section. This approach allows for clear understanding of field namings and their use cases.

Migrate the Ceph nodes section to by-id identifiers¶

Make sure that your MOSK cluster is not currently running an upgrade or any other maintenance process.

Obtain the list of all MiraCeph storage devices that use disk names or disk by-path as identifiers of Ceph node storage devices:

kubectl -n ceph-lcm-mirantis get miraceph -o yaml

Verify the items from the devices sections to be moved to the by-id symlinks. The list of the items to migrate includes:
- A disk name in the name field. For example, sdc, nvme3n1, and so on.
- A disk /dev/disk/by-path symlink in the fullPath field. For example, /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2.
- A disk /dev/disk/by-id symlink in the name field.
- A disk /dev/disk/by-id/wwn symlink, which is programmatically calculated at boot. For example, /dev/disk/by-id/wwn-0x26d546263bd312b8.
For the example above, we have to migrate both items of managed-worker-1, both items of managed-worker-2, and the first item of managed-worker-3. The second item of managed-worker-3 has already been configured in the required format, therefore, we are leaving it as is.
Open the MiraCeph custom resource for editing to start migration of all affected devices items to by-id symlinks:
```
kubectl -n ceph-lcm-mirantis edit miraceph
```

For each affected node from the spec.nodes section, obtain the corresponding status.providerStatus.hardware.storage section in the Machine custom resource located on the management cluster:

kubectl -n <moskClusterProject> get machine <machineName> -o yaml

Substitute <moskClusterProject> with the corresponding cluster namespace and <machineName> with the machine name.

For each affected devices item from the considered Machine, obtain a correct by-id symlink from status.providerStatus.hardware.storage.byIDs. Such by-id symlink must contain status.providerStatus.hardware.storage.serialNumber and must not contain wwn.

For managed-worker-1, according to the example system response above, we can use the following by-id symlinks:
- Replace the first item of devices that contains name: sdc with fullPath: /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_2e52abb48862dbdc
- Replace the second item of devices that contains fullPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2 with fullPath: /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_26d546263bd312b8

Replace all affected devices items in MiraCeph with the obtained ones.

Save and quit editing the MiraCeph custom resource.

After migration, the re-orchestration occurs. The whole procedure should not result in any real changes to the Ceph cluster state in Ceph OSDs.

Migrate the Ceph node groups to by-id identifiers¶

Besides single nodes in the nodes section, your cluster may contain node groups in the nodes section specified with disk names instead of by-id symlinks. Despite of inplace replacement of single nodes storage device identifiers, node groups require another approach because of the repeatable spec section for different nodes.

In the case of migrating storage devices for node groups, use the deviceLabels section to label different disks with the same labels and use these labels in node groups after. For the deviceLabels section specification, refer to Ceph advanced configuration: extraOpts.

The following procedure describes how to keep node groups in the nodes section but use unique by-id identifiers instead of disk names.

To migrate the Ceph node groups to by-id identifiers:

Make sure that your MOSK cluster is not currently running an upgrade or any other maintenance process.

Obtain the list of all MiraCeph storage devices that use disk names or disk by-path as identifiers of Ceph node group storage devices:

kubectl -n ceph-lcm-mirantis get miraceph -o yaml

Verify the items from the devices sections to be moved to by-id symlinks. The list of the items to migrate includes:
- A disk name in the name field. For example, sdc, nvme3n1, and so on.
- A disk /dev/disk/by-path symlink in the fullPath field. For example, /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2.
- A disk /dev/disk/by-id symlink in the name field.
- A disk /dev/disk/by-id/wwn symlink, which is programmatically calculated at boot. For example, /dev/disk/by-id/wwn-0x26d546263bd312b8.
All devices sections in the example above contain disk names in the name field. Therefore, you need to replace them with by-id symlinks.
Open the MiraCeph custom resource for editing to start migration of all affected devices items to by-id symlinks:
```
kubectl -n ceph-lcm-mirantis edit miraceph
```

Within each impacted Ceph node group in the nodes section, add disk labels to the deviceLabels sections for every affected storage device linked with the nodes listed in nodeGroup of that specific node group. Verify that these disk labels are equal to by-id symlinks of corresponding disks.

For example, if the node group rack-1 contains two nodes node-1 and node-2 and spec contains three items with name, you need to obtain proper by-id symlinks for disk names from both nodes and write it down with the same disk labels. The following example contains the labels for by-id symlinks of nvme0n1, nvme1n1, and nvme2n1 disks from node-1 and node-2 correspondingly:

spec:
  extraOpts:
    deviceLabels:
      node-1:
        nvme-1: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543
        nvme-2: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R372150
        nvme-3: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R183266
      node-2:
        nvme-1: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB4040ALR-00007_S46FNY0R900128
        nvme-2: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB4040ALR-00007_S46FNY0R805840
        nvme-3: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB4040ALR-00007_S46FNY0R848469

Note

Keep device labels repeatable for all nodes from the node group. This allows for specifying unified spec for different by-id symlinks of different nodes.

For each affected node group in the nodes section, replace the field with the insufficient disk identifier to the name field with the disk label from the deviceLabels section.

Save and quit editing the MiraCeph custom resource.

After migration, the re-orchestration occurs. The whole procedure should not result in any real changes to the Ceph cluster state in Ceph OSDs.