Searching for results...

No results

Your search did not match anything from Mirantis documentation.
Check your spelling or try different keywords.

An error occurred

An error occurred while using the search.
Try your search again or contact us to let us know about it.

Newer documentation is now live.You are currently reading an older version.

Addressing storage devices using MiraCeph (current)

Warning

This procedure is valid for MOSK clusters that use the MiraCeph custom resource (CR), which is available since MOSK 25.2 to replace the unsupported KaaSCephCluster resource. And MiraCeph will be automatically migrated to CephDeployment in MOSK 26.1. For details, see Deprecation Notes: KaaSCephCluster API on management clusters.

For the equivalent procedure with the unsupported KaaSCephCluster CR, refer to the following section:

Addressing storage devices using KaaSCephCluster (deprecated)

There are several formats to use when specifying and addressing storage devices of a Ceph cluster. The default and recommended one is the /dev/disk/by-id format. This format is reliable and unaffected by the disk controller actions, such as device name shuffling.

Difference between by-id, name, and by-path formats

The storage device /dev/disk/by-id format mostly bases on a disk serial number, which is unique for each disk. A by-id symlink is created by the udev rules in the following format, where <BusID> is an ID of the bus to which the disk is attached and <DiskSerialNumber> stands for a unique disk serial number:

/dev/disk/by-id/<BusID>-<DiskSerialNumber>

Typical by-id symlinks for storage devices look as follows:

/dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543
/dev/disk/by-id/scsi-SATA_HGST_HUS724040AL_PN1334PEHN18ZS
/dev/disk/by-id/ata-WDC_WD4003FZEX-00Z4SA0_WD-WMC5D0D9DMEH

In the example above, symlinks contain the following IDs:

  • Bus IDs: nvme, scsi-SATA and ata

  • Disk serial numbers: SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543, HGST_HUS724040AL_PN1334PEHN18ZS and WDC_WD4003FZEX-00Z4SA0_WD-WMC5D0D9DMEH.

The storage device name format cannot be considered persistent because the sequence in which block devices are added during boot is semi-arbitrary. This means that block device names, for example, nvme0n1 and sdc, are assigned to physical disks during discovery, which may vary inconsistently from the previous node state.

The by-path symlinks are relying on the shortest physical path to the device at boot, so mostly using by-path as an identifier is not prohibited, but Mirantis recommends using by-id symlinks instead.

Therefore, Mirantis highly recommends using storage device by-id symlinks that contain disk serial numbers. This approach enables you to use a persistent device identifier addressed in the Ceph cluster specification.

Example MiraCeph with device by-id identifiers

Below is an example MiraCeph custom resource using fullPath for the /dev/disk/by-id symlink format for storage devices specification:

 apiVersion: lcm.mirantis.com/v1alpha1
 kind: MiraCeph
 metadata:
   name: rook-ceph
   namespace: mosk-ns
 spec:
   nodes:
     # Add the exact node names.
     # Obtain the name from the "kubectl get node" list.
     - name: cz812-mosk-cluster-storage-worker-noefi-58spl
       roles:
       - mgr
       - mon
     # All disk configuration must be reflected in ``status.providerStatus.hardware.storage`` of the ``Machine`` object
       devices:
       - config:
           deviceClass: ssd
         fullPath: /dev/disk/by-id/scsi-1ATA_WDC_WDS100T2B0A-00SM50_200231440912
     - name: cz813-mosk-cluster-storage-worker-noefi-lr4k4
       roles:
       - mgr
       - mon
       devices:
       - config:
           deviceClass: nvme
         fullPath: /dev/disk/by-id/nvme-SAMSUNG_MZ1LB3T8HMLA-00007_S46FNY0R394543
     - name: cz814-mosk-cluster-storage-worker-noefi-z2m67
       roles:
       - mgr
       - mon
       devices:
       - config:
           deviceClass: nvme
         fullPath: /dev/disk/by-id/nvme-SAMSUNG_ML1EB3T8HMLA-00007_S46FNY1R130423
   pools:
   - default: true
     deviceClass: ssd
     name: kubernetes
     replicated:
       size: 3
     role: kubernetes