Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Starting with MOSK 25.2, the MOSK documentation set covers all product layers, including MOSK management (formerly MCC). This means everything you need is in one place. The separate MCC documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Add a Ceph cluster since MOSK 25.2¶

Warning

This procedure is valid for MOSK clusters that use the MiraCeph custom resource (CR), which is available since MOSK 25.2 to replace the deprecated KaaSCephCluster. For the equivalent procedure with the KaaSCephCluster CR, refer to the following section:

Add a Ceph cluster prior to MOSK 25.2

After you add machines to your new MOSK cluster as described in Add a machine, create a Ceph cluster on top of this cluster using CLI.

For an advanced configuration through the MiraCeph CR, see Ceph advanced configuration. For the Ceph Controller configuration through Kubernetes templates to manage Ceph nodes resources, see Enable management of Ceph tolerations and resources.

The procedure below enables you to create a Ceph cluster with minimum three Ceph nodes that provides persistent volumes to the Kubernetes workloads in the MOSK cluster.

To create a Ceph cluster in the MOSK cluster using CLI:

On the management cluster, verify that the MOSK cluster overall status is ready with all conditions in the Ready state:

kubectl -n <moskClusterProject> get cluster <clusterName> -o yaml

Substitute <moskClusterProject> and <clusterName> with the corresponding managed cluster namespace and name accordingly.

Create a YAML file with the Ceph cluster specification:
```
apiVersion: lcm.mirantis.com/v1alpha1
kind: MiraCeph
metadata:
  name: <cephClusterName>
  namespace: ceph-lcm-mirantis
```
Substitute <cephClusterName> with the required name for the Ceph cluster. This name will be used in the Ceph LCM operations.
Add explicit network configuration of the Ceph cluster using the network section:
```
spec:
  network:
    publicNet: <publicNet>
    clusterNet: <clusterNet>
```
Substitute the following values:
- <publicNet> is a CIDR definition or comma-separated list of CIDR definitions (if the MOSK cluster uses multiple networks) of public network for the Ceph data. The values must match the corresponding values of the cluster Subnet object.
- <clusterNet> is a CIDR definition or comma-separated list of CIDR definitions (if the MOSK cluster uses multiple networks) of replication network for the Ceph data. The values must match the corresponding values of the cluster Subnet object.
Configure Ceph Manager and Ceph Monitor roles to select nodes that should place Ceph Monitor and Ceph Manager daemons:
1. On a MOSK cluster, obtain node names to place Ceph Monitor and Ceph Manager daemons at:
```
kubectl get nodes
```
2. Add the nodes section with mon and mgr roles defined:
```
spec:
  nodes:
  - name: <mgr-node-1>
    roles:
    - <role-1>
    - <role-2>
    ...
  - name: <mgr-node-2>
    roles:
    - <role-1>
    - <role-2>
    ...
```
  Substitute <mgr-node-X> with the corresponding node names and <role-X> with the corresponding roles of daemon placement, for example, mon or mgr.
  
  For other optional node parameters, see Ceph advanced configuration.

Configure Ceph OSD daemons for Ceph cluster data storage:

Note

This step involves the deployment of Ceph Monitor and Ceph Manager daemons on nodes that are different from the ones hosting Ceph cluster OSDs. However, it is also possible to colocate Ceph OSDs, Ceph Monitor, and Ceph Manager daemons on the same nodes. You can achieve this by configuring the roles and devices sections accordingly. This kind of configuration flexibility is particularly useful in scenarios such as hyper-converged clusters.

Warning

The minimal production cluster requires at least three nodes for Ceph Monitor daemons and three nodes for Ceph OSDs.

On the management cluster, obtain the names of the machines with disks intended for storing Ceph data:
```
kubectl -n <moskClusterProject> get machine
```

For each machine, use status.providerStatus.hardware.storage to obtain information about node disks:

kubectl -n <moskClusterProject> get machine <machineName> -o yaml

Select by-id symlinks on the disks to be used in the Ceph cluster. The symlinks must meet the following requirements:
- A by-id symlink must contain status.providerStatus.hardware.storage.serialNumber
- A by-id symlink must not contain wwn
For the example above, if you are going to use the sdc disk to store Ceph data on it, use the /dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_2e52abb48862dbdc symlink. It will be persistent and will not be affected by node reboot.

Note

For details about storage device formats, see Addressing storage devices since MOSK 25.2.

For each machine, use status.instanceName to obtain the corresponding node name:

kubectl -n <moskClusterProject> get machine <machineName> -o yaml

Output example of the machine hardware details:

status:
  instanceName: kaas-node-a99016f5-68da-450b-a0c8-06b1f9bb7131

Specify the selected by-id symlinks in the spec.nodes.devices.fullPath field along with the spec.nodes.devices.config.deviceClass field:

Configure the pools for Image, Block Storage, and Compute services.

Note

Ceph validates the specified pools. Therefore, do not omit any of the following pools.
Ceph pool configuration
spec: pools: - default: false deviceClass: hdd name: volumes replicated: size: 3 role: volumes - default: false deviceClass: hdd name: vms replicated: size: 3 role: vms - default: false deviceClass: hdd name: backup replicated: size: 3 role: backup - default: false deviceClass: hdd name: images replicated: size: 3 role: images
Each Ceph pool, depending on its role, has a default targetSizeRatio value that defines the expected consumption of the total Ceph cluster capacity. The default ratio values for MOSK pools are as follows:
- 20.0% for a Ceph pool with role volumes
- 40.0% for a Ceph pool with role vms
- 10.0% for a Ceph pool with role images
- 10.0% for a Ceph pool with role backup
Mirantis highly recommends overriding targetSizeRatio defaults. For details, see Calculate target ratio for Ceph pools.

Configure Ceph Block Pools to use RBD. For the detailed configuration, refer to Pool parameters. Example configuration:

spec:
  pools:
   - default: true
     deviceClass: hdd
     name: kubernetes
     replicated:
       size: 3
     role: kubernetes

Configure Ceph Object Storage to use OpenStack Swift Object Storage.

Optional. Configure Ceph Shared Filesystem to use CephFS. For the detailed configuration, refer to Configure Ceph Shared File System (CephFS).

When the Ceph cluster specification is complete, apply the built YAML file on the MOSK cluster:

kubectl apply -f <miraceph-template>.yaml

Substitue <miraceph-template> with the name of the file containing the MiraCeph specification.

Wait for the MiraCephHealth object to be created and then for status.shortClusterInfo.state to become Ready:
```
kubectl -n ceph-lcm-mirantis get mchealth -o yaml
```
Verify the cluster as described in Verify Ceph.
Once all pools are created, verify that an appropriate secret required for a successful deployment of the OpenStack services that rely on Ceph is created in the openstack-ceph-shared namespace:
```
kubectl -n openstack-ceph-shared get secrets openstack-ceph-keys
```
Example of a positive system response:
```
NAME                  TYPE     DATA   AGE
openstack-ceph-keys   Opaque   7      36m
```