Add a Ceph cluster using CLI¶

This section explains how to create a Ceph cluster on top of a managed cluster using the Mirantis Container Cloud CLI. As a result, you will deploy a Ceph cluster with minimum three Ceph nodes that provide persistent volumes to the Kubernetes workloads for your managed cluster.

Note

For the advanced configuration through the KaaSCephCluster custom resource, see Ceph advanced configuration.

For the configuration of the Ceph Controller through Kubernetes templates to manage Ceph node resources, see Enable Ceph tolerations and resources management.

To create a Ceph cluster in a managed cluster:

Verify that the managed cluster overall status is ready with all conditions in the Ready state:

kubectl -n <managedClusterProject> get cluster <clusterName> -o yaml

Substitute <managedClusterProject> and <clusterName> with the corresponding managed cluster namespace and name accordingly.

Example output:

status:
  providerStatus:
    ready: true
    conditions:
    - message: Helm charts are successfully installed(upgraded).
      ready: true
      type: Helm
    - message: Kubernetes objects are fully up.
      ready: true
      type: Kubernetes
    - message: All requested nodes are ready.
      ready: true
      type: Nodes
    - message: Maintenance state of the cluster is false
      ready: true
      type: Maintenance
    - message: TLS configuration settings are applied
      ready: true
      type: TLS
    - message: Kubelet is Ready on all nodes belonging to the cluster
      ready: true
      type: Kubelet
    - message: Swarm is Ready on all nodes belonging to the cluster
      ready: true
      type: Swarm
    - message: All provider instances of the cluster are Ready
      ready: true
      type: ProviderInstance
    - message: LCM agents have the latest version
      ready: true
      type: LCMAgent
    - message: StackLight is fully up.
      ready: true
      type: StackLight
    - message: OIDC configuration has been applied.
      ready: true
      type: OIDC
    - message: Load balancer 10.100.91.150 for kubernetes API has status HEALTHY
      ready: true
      type: LoadBalancer

Create a YAML file with the Ceph cluster specification:

apiVersion: kaas.mirantis.com/v1alpha1
kind: KaaSCephCluster
metadata:
  name: <cephClusterName>
  namespace: <managedClusterProject>
spec:
  k8sCluster:
    name: <clusterName>
    namespace: <managedClusterProject>

Substitute <cephClusterName> with the desired name for the Ceph cluster. This name will be used in the Ceph LCM operations.

Select from the following options:
- Add explicit network configuration of the Ceph cluster using the network section:
```
spec:
  cephClusterSpec:
    network:
      publicNet: <publicNet>
      clusterNet: <clusterNet>
```
  Substitute the following values:
  - <publicNet> is a CIDR definition or comma-separated list of CIDR definitions (if the managed cluster uses multiple networks) of public network for the Ceph data. The values should match the corresponding values of the cluster Subnet object.
  - <clusterNet> is a CIDR definition or comma-separated list of CIDR definitions (if the managed cluster uses multiple networks) of replication network for the Ceph data. The values should match the corresponding values of the cluster Subnet object.
- Configure Subnet objects for the Storage access network by setting ipam/SVC-ceph-public: "1" and ipam/SVC-ceph-cluster: "1" labels to the corresponding Subnet objects. For more details, refer to Create subnets for a managed cluster using CLI, Step 5.
Configure Ceph Manager and Ceph Monitor roles to select nodes that should place Ceph Monitor and Ceph Manager daemons:
1. Obtain the names of the machines to place Ceph Monitor and Ceph Manager daemons at:
```
kubectl -n <managedClusterProject> get machine
```
2. Add the nodes section with mon and mgr roles defined:
```
spec:
  cephClusterSpec:
    nodes:
      <mgr-node-1>:
        roles:
        - <role-1>
        - <role-2>
        ...
      <mgr-node-2>:
        roles:
        - <role-1>
        - <role-2>
        ...
```
  Substitute <mgr-node-X> with the corresponding Machine object names and <role-X> with the corresponding roles of daemon placement, for example, mon or mgr.
  
  See also
  
  Node parameters

Configure Ceph OSD daemons for Ceph cluster data storage:

Note

This step involves the deployment of Ceph Monitor and Ceph Manager daemons on nodes that are different from the ones hosting Ceph cluster OSDs. However, it is also possible to colocate Ceph OSDs, Ceph Monitor, and Ceph Manager daemons on the same nodes. You can achieve this by configuring the roles and storageDevices sections accordingly. This kind of configuration flexibility is particularly useful in scenarios such as hyper-converged clusters.

Warning

The minimal production cluster requires at least three nodes for Ceph Monitor daemons and three nodes for Ceph OSDs.

Obtain the names of the machines with disks intended for storing Ceph data:
```
kubectl -n <managedClusterProject> get machine
```

For each machine, use status.providerStatus.hardware.storage to obtain information about node disks:

kubectl -n <managedClusterProject> get machine <machineName> -o yaml

Output example of the machine hardware details:

status:
  providerStatus:
    hardware:
      storage:
      - byID: /dev/disk/by-id/wwn-0x05ad99618d66a21f
        byIDs:
        - /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_05ad99618d66a21f
        - /dev/disk/by-id/scsi-305ad99618d66a21f
        - /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_05ad99618d66a21f
        - /dev/disk/by-id/wwn-0x05ad99618d66a21f
        byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:0
        byPaths:
        - /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:0
        name: /dev/sda
        serialNumber: 05ad99618d66a21f
        size: 61
        type: hdd
      - byID: /dev/disk/by-id/wwn-0x26d546263bd312b8
        byIDs:
        - /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_26d546263bd312b8
        - /dev/disk/by-id/scsi-326d546263bd312b8
        - /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_26d546263bd312b8
        - /dev/disk/by-id/wwn-0x26d546263bd312b8
        byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2
        byPaths:
        - /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:2
        name: /dev/sdb
        serialNumber: 26d546263bd312b8
        size: 32
        type: hdd
      - byID: /dev/disk/by-id/wwn-0x2e52abb48862dbdc
        byIDs:
        - /dev/disk/by-id/lvm-pv-uuid-MncrcO-6cel-0QsB-IKaY-e8UK-6gDy-k2hOtf
        - /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_2e52abb48862dbdc
        - /dev/disk/by-id/scsi-32e52abb48862dbdc
        - /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_2e52abb48862dbdc
        - /dev/disk/by-id/wwn-0x2e52abb48862dbdc
        byPath: /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:1
        byPaths:
        - /dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:0:1
        name: /dev/sdc
        serialNumber: 2e52abb48862dbdc
        size: 61
        type: hdd

Select by-id symlinks on the disks to be used in the Ceph cluster. The symlinks should meet the following requirements:
- A by-id symlink should contain status.providerStatus.hardware.storage.serialNumber
- A by-id symlink should not contain wwn
For the example above, if you are willing to use the sdc disk to store Ceph data on it, use the /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_2e52abb48862dbdc symlink. It will be persistent and will not be affected by node reboot.

See also

Addressing storage devices

Sepcify by-id symlinks:

Since Container Cloud 2.25.0

Specify selected by-id symlinks in the spec.cephClusterSpec.nodes.storageDevices.fullPath field along with the spec.cephClusterSpec.nodes.storageDevices.config.deviceClass field:

spec:
  cephClusterSpec:
    nodes:
      <storage-node-1>:
        storageDevices:
        - fullPath: <byIDSymlink-1>
          config:
            deviceClass: <deviceClass-1>
        - fullPath: <byIDSymlink-2>
          config:
            deviceClass: <deviceClass-1>
        - fullPath: <byIDSymlink-3>
          config:
            deviceClass: <deviceClass-2>
        ...
      <storage-node-2>:
        storageDevices:
        - fullPath: <byIDSymlink-4>
          config:
            deviceClass: <deviceClass-1>
        - fullPath: <byIDSymlink-5>
          config:
            deviceClass: <deviceClass-1>
        - fullPath: <byIDSymlink-6>
          config:
            deviceClass: <deviceClass-2>
      <storage-node-3>:
        storageDevices:
        - fullPath: <byIDSymlink-7>
          config:
            deviceClass: <deviceClass-1>
        - fullPath: <byIDSymlink-8>
          config:
            deviceClass: <deviceClass-1>
        - fullPath: <byIDSymlink-9>
          config:
            deviceClass: <deviceClass-2>

Substitute the following values:

<storage-node-X> with the corresponding Machine object names
<byIDSymlink-X> with the obtained by-id symlinks from status.providerStatus.hardware.storage.byIDs
<deviceClass-X> with the obtained disk types from status.providerStatus.hardware.storage.type

Before Container Cloud 2.25.0

Specify selected by-id symlinks in the spec.cephClusterSpec.nodes.storageDevices.name field along with the spec.cephClusterSpec.nodes.storageDevices.config.deviceClass field:

spec:
  cephClusterSpec:
    nodes:
      <storage-node-1>:
        storageDevices:
        - name: <byIDSymlink-1>
          config:
            deviceClass: <deviceClass-1>
        - name: <byIDSymlink-2>
          config:
            deviceClass: <deviceClass-1>
        - name: <byIDSymlink-3>
          config:
            deviceClass: <deviceClass-2>
        ...
      <storage-node-2>:
        storageDevices:
        - name: <byIDSymlink-4>
          config:
            deviceClass: <deviceClass-1>
        - name: <byIDSymlink-5>
          config:
            deviceClass: <deviceClass-1>
        - name: <byIDSymlink-6>
          config:
            deviceClass: <deviceClass-2>
      <storage-node-3>:
        storageDevices:
        - name: <byIDSymlink-7>
          config:
            deviceClass: <deviceClass-1>
        - name: <byIDSymlink-8>
          config:
            deviceClass: <deviceClass-1>
        - name: <byIDSymlink-9>
          config:
            deviceClass: <deviceClass-2>

Substitute the following values:

<storage-node-X> with the corresponding Machine object names
<byIDSymlink-X> with the obtained by-id symlinks from status.providerStatus.hardware.storage.byIDs
<deviceClass-X> with the obtained disk types from status.providerStatus.hardware.storage.type

Optional. Configure Ceph Block Pools to use RBD. For the detailed configuration, refer to Pool parameters.

Example configuration:

spec:
  cephClusterSpec:
    pools:
    - name: kubernetes
      role: kubernetes
      deviceClass: hdd
      replicated:
        size: 3
        targetSizeRatio: 10.0
      default: true

Optional. Configure Ceph Object Storage to use RGW. For the detailed configuration, refer to RADOS Gateway parameters.

Example configuration:

spec:
  cephClusterSpec:
    objectStorage:
      rgw:
        dataPool:
          deviceClass: hdd
          erasureCoded:
            codingChunks: 1
            dataChunks: 2
          failureDomain: host
        gateway:
          instances: 3
          port: 80
          securePort: 8443
        metadataPool:
          deviceClass: hdd
          failureDomain: host
          replicated:
            size: 3
        name: object-store
        preservePoolsOnDelete: false

Optional. Configure Ceph Shared Filesystem to use CephFS. For the detailed configuration, refer to Enable Ceph Shared File System (CephFS).

Example configuration:

spec:
  cephClusterSpec:
    sharedFilesystem:
      cephFS:
      - name: cephfs-store
        dataPools:
        - name: cephfs-pool-1
          deviceClass: hdd
          replicated:
            size: 3
          failureDomain: host
        metadataPool:
          deviceClass: nvme
          replicated:
            size: 3
          failureDomain: host
        metadataServer:
          activeCount: 1
          activeStandby: false

When the Ceph cluster specification is complete, apply the built YAML file on the management cluster:

kubectl apply -f <kcc-template>.yaml

Substitue <kcc-template> with the name of the file containing the KaaSCephCluster specification.

Wait for the KaaSCephCluster status and then for status.shortClusterInfo.state to become Ready:
```
kubectl -n <managedClusterProject> get kcc -o yaml
```