Configure Ceph disks in a host profile¶

This section describes how to configure devices for the Ceph cluster in the BareMetalHostProfile object of a MOSK cluster.

To configure disks for a Ceph cluster:

Open the BareMetalHostProfile object of a MOSK cluster for editing.

In the spec.devices section, add each disk intended for use as a Ceph OSD data device with size: 0 and wipe: true.

Example configuration for sde - sdh disks to use as Ceph OSDs:

spec:
  devices:
  ...
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:1
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:3
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:4
      size: 0
      wipe: true

If you plan to use a separate metadata device for Ceph OSD, configure the spec.devices section as described below.

Important

Mirantis highly recommends configuring disk partitions for Ceph OSD metadata using BareMetalHostProfile.

Configuration of a separate metadata device for Ceph OSD

Add the device to spec.devices with a single partition that will use the entire disk size.

For example, if you plan to use four Ceph OSDs with a separate metadata device for each Ceph OSD, configure the spec.devices section as follows:

spec:
  devices:
  ...
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:5
      wipe: true
    partitions:
    - name: ceph_meta
      size: 0
      wipe: true

Create a volume group on top of the defined partition and create the required number of logical volumes (LVs) on top of the created volume group (VG). Add one logical volume per one Ceph OSD on the node.

Example snippet of an LVM configuration for a Ceph metadata disk:

spec:
  ...
  volumeGroups:
  ...
  - devices:
    - partition: ceph_meta
    name: bluedb
  logicalVolumes:
  ...
  - name: meta_1
    size: 25%VG
    vg: bluedb
  - name: meta_2
    size: 25%VG
    vg: bluedb
  - name: meta_3
    size: 25%VG
    vg: bluedb
  - name: meta_4
    size: 25%VG
    vg: bluedb

Important

Plan LVs of a separate metadata device thoroughly. Any logical volume misconfiguration causes redeployment of all Ceph OSDs that use this disk as metadata devices.

Note

General Ceph recommendation is to have a metadata device in between 1% to 4% of the Ceph OSD data size. Mirantis highly recommends having at least 4% of Ceph OSD data size.

If you plan using a disk as a separate metadata device for 10 Ceph OSDs, define the size of an LV for each Ceph OSD in between 1% to 4% of the corresponding Ceph OSD data size. If RADOS Gateway is enabled, the minimum data size must be 4%. For details, see Ceph documentation: Bluestore config reference.

For example, if the total data size of 10 Ceph OSDs equals 1Tb with 100Gb each, assign a metadata disk less than 10Gb with 1Gb per each LV. The recommended size is 40Gb with 4Gb per each LV.

After applying BareMetalHostProfile, the bare metal provider creates an LVM partitioning for the metadata disk and places these volumes as /dev paths, for example, /dev/bluedb/meta_1 or /dev/bluedb/meta_3.

Example template of a host profile configuration for Ceph

spec:
  ...
  devices:
  ...
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:1
      wipe: true
  - device:
      byName: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:3
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:4
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:5
      wipe: true
    partitions:
    - name: ceph_meta
      size: 0
      wipe: true
  volumeGroups:
  ...
  - devices:
    - partition: ceph_meta
    name: bluedb
  logicalVolumes:
  ...
  - name: meta_1
    size: 25%VG
    vg: bluedb
  - name: meta_2
    size: 25%VG
    vg: bluedb
  - name: meta_3
    size: 25%VG
    vg: bluedb
  - name: meta_4
    size: 25%VG
    vg: bluedb

After applying such BareMetalHostProfile to a node, the nodes spec of the CephDeployment object contains the following storageDevices section:

spec:
  ...
  nodes:
  - name: storage-worker-1
    ...
    devices:
    - config:
        deviceClass: hdd
        metadataDevice: /dev/bluedb/meta_1
      fullPath: /dev/disk/by-id/scsi-0ATA_ST4000NM002A-2HZ101_WS20NEGE
    - config:
        deviceClass: hdd
        metadataDevice: /dev/bluedb/meta_2
      fullPath: /dev/disk/by-id/scsi-0ATA_ST4000NM002A-2HZ_WS20LEL3
    - config:
        deviceClass: hdd
        metadataDevice: /dev/bluedb/meta_3
      fullPath: /dev/disk/by-id/ata-HGST_HUS724040ALA640_PN1334PEDN9SSU
    - config:
        deviceClass: hdd
        metadataDevice: /dev/bluedb/meta_4
      fullPath: /dev/disk/by-id/scsi-0ATA_HGST_HUS724040ALA640_PN1334PEDN9SSU

No results

An error occurred

Configure Ceph disks in a host profile¶