Configure Ceph Shared File System (CephFS)
The Ceph Shared File System, or CephFS, provides the ability to create
read/write shared file system Persistent Volumes (PVs). These PVs support the
ReadWriteMany access mode for the FileSystem volume mode.
CephFS deploys its own daemons called MetaData Servers or Ceph MDS. For
details, see Ceph Documentation: Ceph File System.
Note
By design, the default CephFS data and metadata pools must be replicated only.
CephFS specification parameters
The CephDeployment custom resource (CR) spec includes the sharedFilesystem.cephFilesystems section
with the following CephFS parameters:
name- CephFS instance name.spec- CephFS spec definition based on the Rook API CephFS specification.
For configuration details, refer to the CephFilesystems parameters.
Example of the CephFS specification:
spec:
sharedFilesystem:
cephFilesystems:
- name: cephfs-store
spec:
dataPools:
- name: default-pool
deviceClass: ssd
replicated:
size: 3
failureDomain: host
- name: second-pool
deviceClass: hdd
failureDomain: rack
erasureCoded:
dataChunks: 2
codingChunks: 1
metadataPool:
deviceClass: nvme
replicated:
size: 3
failureDomain: host
metadataServer:
activeCount: 1
activeStandby: false
resources: # example, non-prod values
requests:
memory: 1Gi
cpu: 1
limits:
memory: 2Gi
cpu: 2
Warning
We do not recommend using replicated Ceph pools with replicated.size of less
than 3 in production environments. For such pools, Ceph OSD removal may cause
issues. If replicated.size is 1, Ceph LCM operations cannot function correctly.
Set replicated.size to 3 for each replicated Ceph pool. For erasure-coded pools,
the minimal recommended configuration is 2+1, that is,
erasureCoded.dataChunks: 2 and erasureCoded.codingChunks: 1.
Warning
Modifying dataPools or metadataPool on a deployed CephFS has no effect. You can manually adjust pool settings
through the Ceph CLI. However, for any changes in dataPools or metadataPool, we recommend re-creating CephFS.
Configure CephFS
Optional. Verify that the CephFS CSI driver is enabled. Rook enables it when the
rook.rookConfig.csiCephFsEnabledHelm chart value istrue, which is the default. If you have disabled it, re-enable it by upgrading the Pelagia Helm release:helm upgrade --install pelagia-ceph oci://registry.mirantis.com/pelagia/pelagia-ceph --version ${PELAGIA_VERSION} -n ceph-lcm-mirantis \ --set rook.rookConfig.csiCephFsEnabled=true
Rook will enable the CephFS CSI driver, which deploys the CephFS CSI node and controller plugins.
Open the
CephDeploymentCR for editing:kubectl -n ceph-lcm-mirantis edit cephdpl
Update the
sharedFilesystemsection specification as required using the configuration reference above. For example:spec: sharedFilesystem: cephFilesystems: - name: cephfs-store spec: dataPools: - name: cephfs-pool-1 deviceClass: hdd replicated: size: 3 failureDomain: host metadataPool: deviceClass: nvme replicated: size: 3 failureDomain: host metadataServer: activeCount: 1 activeStandby: false
Define the
mdsrole for the corresponding nodes where Ceph MDS daemons should be deployed. We recommend labeling only one node with themdsrole. For example:spec: nodes: ... - name: worker-1 roles: ... - mds
Once CephFS is specified in the CephDeployment CR, Pelagia Deployment Controller will validate it and
request Rook to create CephFS. Then Pelagia Deployment Controller will create a Kubernetes StorageClass,
required to start provisioning the storage, which will operate the CephFS CSI driver to create Kubernetes PVs.