Configure Ceph Shared File System (CephFS)

The Ceph Shared File System, or CephFS, provides the ability to create read/write shared file system Persistent Volumes (PVs). These PVs support the ReadWriteMany access mode for the FileSystem volume mode. CephFS deploys its own daemons called MetaData Servers or Ceph MDS. For details, see Ceph Documentation: Ceph File System.

Note

By design, the default CephFS data and metadata pools must be replicated only.

CephFS specification parameters

The CephDeployment custom resource (CR) spec includes the sharedFilesystem.cephFilesystems section with the following CephFS parameters:

  • name - CephFS instance name.

  • spec - CephFS spec definition based on the Rook API CephFS specification.

For configuration details, refer to the CephFilesystems parameters.

Example of the CephFS specification:

spec:
  sharedFilesystem:
    cephFilesystems:
    - name: cephfs-store
      spec:
        dataPools:
        - name: default-pool
          deviceClass: ssd
          replicated:
            size: 3
          failureDomain: host
        - name: second-pool
          deviceClass: hdd
          failureDomain: rack
          erasureCoded:
            dataChunks: 2
            codingChunks: 1
        metadataPool:
          deviceClass: nvme
          replicated:
            size: 3
          failureDomain: host
        metadataServer:
          activeCount: 1
          activeStandby: false
          resources: # example, non-prod values
            requests:
              memory: 1Gi
              cpu: 1
            limits:
              memory: 2Gi
              cpu: 2

Warning

We do not recommend using replicated Ceph pools with replicated.size of less than 3 in production environments. For such pools, Ceph OSD removal may cause issues. If replicated.size is 1, Ceph LCM operations cannot function correctly. Set replicated.size to 3 for each replicated Ceph pool. For erasure-coded pools, the minimal recommended configuration is 2+1, that is, erasureCoded.dataChunks: 2 and erasureCoded.codingChunks: 1.

Warning

Modifying dataPools or metadataPool on a deployed CephFS has no effect. You can manually adjust pool settings through the Ceph CLI. However, for any changes in dataPools or metadataPool, we recommend re-creating CephFS.

Configure CephFS

  1. Optional. Verify that the CephFS CSI driver is enabled. Rook enables it when the rook.rookConfig.csiCephFsEnabled Helm chart value is true, which is the default. If you have disabled it, re-enable it by upgrading the Pelagia Helm release:

    helm upgrade --install pelagia-ceph oci://registry.mirantis.com/pelagia/pelagia-ceph --version ${PELAGIA_VERSION} -n ceph-lcm-mirantis \
         --set rook.rookConfig.csiCephFsEnabled=true
    

    Rook will enable the CephFS CSI driver, which deploys the CephFS CSI node and controller plugins.

  2. Open the CephDeployment CR for editing:

    kubectl -n ceph-lcm-mirantis edit cephdpl
    
  3. Update the sharedFilesystem section specification as required using the configuration reference above. For example:

    spec:
      sharedFilesystem:
        cephFilesystems:
        - name: cephfs-store
          spec:
            dataPools:
            - name: cephfs-pool-1
              deviceClass: hdd
              replicated:
                size: 3
              failureDomain: host
            metadataPool:
              deviceClass: nvme
              replicated:
                size: 3
              failureDomain: host
            metadataServer:
              activeCount: 1
              activeStandby: false
    
  4. Define the mds role for the corresponding nodes where Ceph MDS daemons should be deployed. We recommend labeling only one node with the mds role. For example:

    spec:
      nodes:
      ...
      - name: worker-1
        roles:
        ...
        - mds
    

Once CephFS is specified in the CephDeployment CR, Pelagia Deployment Controller will validate it and request Rook to create CephFS. Then Pelagia Deployment Controller will create a Kubernetes StorageClass, required to start provisioning the storage, which will operate the CephFS CSI driver to create Kubernetes PVs.