Ceph advanced configuration

Ceph advanced configuration

This section describes how to configure a Ceph cluster through the KaaSCephCluster (kaascephclusters.kaas.mirantis.com) CR during or after the deployment of a management or managed cluster.

The KaaSCephCluster CR spec has two sections, cephClusterSpec and k8sCluster and specifies the nodes to deploy as Ceph components. Based on the roles definitions in the KaaSCephCluster CR, Ceph Controller automatically labels nodes for Ceph Monitors and Managers. Ceph OSDs are deployed based on the storageDevices parameter defined for each Ceph node.

For a default KaaSCephCluster CR, see templates/bm/kaascephcluster.yaml.template. For details on how to configure the default template for a baremetal-based cluster bootstrap, see Deployment Guide: Bootstrap a management cluster.

To configure a Ceph cluster:

  1. Select from the following options:

    • If you do not have a management cluster yet, open kaascephcluster.yaml.template for editing.

    • If the management cluster is already deployed, open the KaasCephCluster CR for editing:

      • If the Ceph cluster is placed in the management cluster:

        kubectl edit kaascephcluster
        
      • If the Ceph cluster is placed in a managed cluster:

        kubectl edit kaascephcluster -n <managedClusterProjectName>
        

        Substitute <managedClusterProjectName> with a corresponding value.

  2. Using the tables below, configure the Ceph cluster as required.

    High-level parameters

    Parameter

    Description

    cephClusterSpec

    Describes a Ceph cluster in the management cluster. For details on cephClusterSpec parameters, see the tables below.

    k8sCluster

    Defines the management cluster on which the KaaSCephCluster depends on. Use the k8sCluster parameter if the name or namespace of the management cluster differs from default one:

    spec:
      k8sCluster:
        name: kaas-mgmt
        namespace: default
    
    General parameters

    Parameter

    Description

    manageOsds

    Recommended. Enables automated management of Ceph OSDs. For details, see Enable automated Ceph LCM.

    clusterNet

    Specifies the CIDR for Ceph OSD replication network.

    publicNet

    Specifies the CIDR for communication between the service and operator.

    nodes

    Specifies the list of Ceph nodes. For details, see Node parameters. The nodes parameter is a map with machine names as keys and Ceph node specifications as values, for example:

    nodes:
      master-0:
        <node spec>
      master-1:
        <node spec>
      ...
      worker-0:
        <node spec>
    

    pools

    Specifies the list of Ceph pools. For details, see Pool parameters.

    objectStorage Available since 2.6.0

    Specifies the parameters for Object Storage, such as RADOS Gateway, the Ceph Object Storage. Starting from Container Cloud 2.7.0, also specifies the RADOS Gateway Multisite configuration. For details, see RADOS Gateway parameters and Multisite parameters.

    rgw Deprecated since 2.6.0

    Specifies RADOS Gateway, the Ceph Object Storage. For details, see RADOS Gateway parameters.

    rgw Deprecated since 2.6.0

    Specifies RADOS Gateway, the Ceph Object Storage. For details, see RADOS Gateway parameters.

    maintenance Available since 2.6.0

    Enables or disables the noout, norebalance, and nofill flags on the entire Ceph cluster. Set to false by default. Mirantis strongly recommends not using it on production deployments other than during update.

    Example configuration:

    spec:
      cephClusterSpec:
        manageOsds: true
        network:
          clusterNet: 10.10.10.0/24
          publicNet: 10.10.11.0/24
        nodes:
          master-0:
            <node spec>
          ...
        pools:
        - <pool spec>
        ...
    
    Node parameters

    Parameter

    Description

    roles

    Specifies the mon or mgr daemon to be installed on a Ceph node. You can place the daemons on any nodes upon your decision. Consider the following recommendations:

    • The recommended number of Ceph Monitors in a Ceph cluster is 3. Therefore, at least 3 Ceph nodes must contain the mon item in the roles parameter.

    • The number of Ceph Monitors must be odd. For example, if the KaaSCephCluster spec contains 3 Ceph monitors and you need to add more, the number of Ceph monitors must equal 5, 7, 9, and so on.

    • Do not add more than 2 Ceph monitors at a time and wait until the Ceph cluster is Ready.

    • For a better HA, the number of mgr roles must equal the number of mon roles.

      If a Ceph node contains a mon role, the Ceph Monitor Pod will be deployed on it. If the Ceph node contains a mgr role, it informs the Ceph Controller that a Ceph Manager can be deployed on that node. However, only one Ceph Manager must be deployed on a node.

    storageDevices

    Specifies the list of devices to use for Ceph OSD deployment. Includes the following parameters:

    • name - the device name placed in the /dev folder. For example, vda.

    • config - a map of device configurations that must contain a device class. The device class must be defined in a pool and can contain a metadata device:

      storageDevices:
      - name: sdc
        config:
          deviceClass: hdd
          metadataDevice: nvme01
      

      The underlying storage format to use for Ceph OSDs is BlueStore.

    crush Available since 2.7.0

    Specifies the explicit key-value CRUSH topology for a node. For details, see Ceph official documentation: CRUSH maps. Includes the following parameters:

    • datacenter - a physical data center that consists of rooms and handles data.

    • room - a room that accommodates one or more racks with hosts.

    • pdu - a power distribution unit (PDU) device that has multiple outputs and distributes electric power to racks located within a data center.

    • row - a row of computing racks inside room.

    • rack - a computing rack that accommodates one or more hosts.

    • chassis - a bare metal structure that houses or physically assembles hosts.

    • region - the geographic location of one or more Ceph Object instances within one or more zones.

    • zone - a logical group that consists of one or more Ceph Object instances.

    Example configuration:

    crush:
      datacenter: dc1
      room: room1
      pdu: pdu1
      row: row1
      rack: rack1
      chassis: ch1
      region: region1
      zone: zone1
    
    Pool parameters

    Parameter

    Description

    name

    Specifies the pool name as a prefix for each Ceph block pool.

    role

    Specifies the pool role and is used mostly for Mirantis OpenStack for Kubernetes (MOS) pools.

    default

    Defines if the pool and dependent StorageClass should be set as default. Must be enabled only for one pool.

    deviceClass

    Specifies the device class for the defined pool. Possible values are HDD, SSD, and NVMe.

    replicated

    The number of pool replicas. The replicated parameter is mutually exclusive with erasureCoded.

    erasureCoded

    Enables the erasure-coded pool. For details, see Rook documentation: Erasure coded and Ceph documentation: Erasure coded pool. The erasureCoded parameter is mutually exclusive with replicated.

    failureDomain

    The failure domain across which the replicas or chunks of data will be spread. Set to host by default. The possible values are osd or host.

    Example configuration:

    pools:
    - name: kubernetes
      role: kubernetes
      deviceClass: hdd
      replicated:
        size: 3
      default: true
    

    To configure additional required pools for MOS, see MOS Deployment Guide: Deploy a Ceph cluster.

    RADOS Gateway parameters

    Parameter

    Description

    name

    Ceph Object Storage instance name.

    dataPool

    Mutually exclusive with the zone parameter. Object storage data pool spec that should only contain replicated or erasureCoded and failureDomain parameters. The failureDomain parameter may be set to osd or host, defining the failure domain across which the data will be spread. For dataPool, Mirantis recommends using an erasureCoded pool. For details, see Rook documentation: Erasure coding. For example:

    cephClusterSpec:
      objectStorage:
        rgw:
          dataPool:
            erasureCoded:
              codingChunks: 1
              dataChunks: 2
    

    metadataPool

    Mutually exclusive with the zone parameter. Object storage metadata pool spec that should only contain replicated and failureDomain parameters. The failureDomain parameter may be set to osd or host, defining the failure domain across which the data will be spread. Can use only replicated settings. For example:

    cephClusterSpec:
      objectStorage:
        rgw:
          metadataPool:
            replicated:
              size: 3
            failureDomain: host
    

    where replicated.size is the number of full copies of data on multiple nodes.

    gateway

    The gateway settings corresponding to the rgw daemon settings. Includes the following parameters:

    • port - the port on which the Ceph RGW service will be listening on HTTP.

    • securePort - the port on which the Ceph RGW service will be listening on HTTPS.

    • instances - the number of pods in the Ceph RGW ReplicaSet. If allNodes is set to true, a DaemonSet is created instead.

      Note

      Mirantis recommends using 2 instances for Ceph Object Storage.

    • allNodes - defines whether to start the Ceph RGW pods as a DaemonSet on all nodes. The instances parameter is ignored if allNodes is set to true.

    For example:

    cephClusterSpec:
      objectStorage:
        rgw:
          gateway:
            allNodes: false
            instances: 1
            port: 80
            securePort: 8443
    

    preservePoolsOnDelete

    Defines whether to delete the data and metadata pools in the rgw section if the object storage is deleted. Set this parameter to true if you need to store data even if the object storage is deleted. However, Mirantis recommends setting this parameter to false.

    users and buckets

    Optional. To create new Ceph RGW resources, such as buckets or users, specify the following keys. Ceph controller will automatically create the specified object storage users and buckets in the Ceph cluster.

    • users - a list of strings that contain user names to create for object storage.

    • buckets - a list of strings that contain bucket names to create for object storage.

    zone Available since 2.7.0

    Optional. Mutually exclusive with metadataPool and dataPool. Defines the Ceph Multisite zone where the object storage must be placed. Includes the name parameter that must be set to one of the zones items. For details, see Enable Multisite for Ceph RGW Object Storage.

    For example:

    cephClusterSpec:
      objectStorage:
        multisite:
          zones:
          - name: master-zone
          ...
        rgw:
          zone:
            name: master-zone
    

    For configuration example, see Enable Ceph RGW Object Storage.

    Multisite parameters

    Parameter

    Description

    realms Available since 2.7.0, Technical Preview

    List of realms to use, represents the realm namespaces. Includes the following parameters:

    • name - the realm name.

    • pullEndpoint - optional, required only when the master zone is in a different storage cluster. The endpoint, access key, and system key of the system user from the realm to pull from. Includes the following parameters:

      • endpoint - the endpoint of the master zone in the master zone group.

      • accessKey - the access key of the system user from the realm to pull from.

      • secretKey - the system key of the system user from the realm to pull from.

    zoneGroups Available since 2.7.0, Technical Preview

    The list of zone groups for realms. Includes the following parameters:

    • name - the zone group name.

    • realmName - the realm namespace name to which the zone group belongs to.

    zones Available since 2.7.0, Technical Preview

    The list of zones used within one zone group. Includes the following parameters:

    • name - the zone name.

    • metadataPool - the settings used to create the Object Storage metadata pools. Must use replication. For details, see Pool parameters.

    • dataPool - the settings to create the Object Storage data pool. Can use replication or erasure coding. For details, see Pool parameters.

    • zoneGroupName - the zone group name.

    For configuration example, see Enable Multisite for Ceph RGW Object Storage.

  3. Select from the following options:

    • If you are bootstrapping a management cluster, save the updated KaaSCephCluster template to the templates/bm/kaascephcluster.yaml.template file and proceed with the bootstrap.

    • If you are creating a managed cluster, save the updated KaaSCephCluster template to the corresponding file and proceed with the managed cluster creation.

    • If you are configuring KaaSCephCluster of an existing management cluster, run the following command:

      kubectl apply
      
    • If you are configuring KaaSCephCluster of an existing managed cluster, run the following command:

      kubectl apply -n <managedClusterProjectName>
      

      Substitute <managedClusterProjectName> with the corresponding value.