Enable Ceph multinetwork

Ceph allows establishing multiple IP networks and subnet masks for clusters with configured L3 network rules. In Container Cloud, you can configure multinetwork through the network section of the KaaSCephCluster CR. Ceph controller uses this section to specify the Ceph networks for external access and internal daemon communication. The parameters in the network section use the CIDR notation, for example, 10.0.0.0/24.

Before enabling multiple networks for a Ceph cluster, consider the following requirements:

  • Do not confuse the IP addresses you define with the public-facing IP addresses the network clients may use to access the services.

  • If you define more than one IP address and subnet mask for the public or cluster network, ensure that the subnets within the network can route to each other.

  • Include each IP address or subnet in the network section to IP tables and open ports for them as necessary.

  • The pods of the Ceph OSD and RadosGW daemons use cross-pods health checkers to verify that the entire Ceph cluster is healthy. Therefore, each CIDR must be accessible inside Ceph pods.

  • Avoid using the 0.0.0.0/0 CIDR in the network section. With a zero range in publicNet and/or clusterNet, the Ceph daemons behavior is unpredictable.

To enable multinetwork for Ceph:

  1. Select from the following options:

    • If you do not have a management cluster yet, open kaascephcluster.yaml.template for editing.

    • If the management cluster is already deployed, open the KaasCephCluster CR for editing. Select from the following options:

      • If the Ceph cluster is placed in the management cluster:

        kubectl edit kaascephcluster
        
      • If the Ceph cluster is placed in a managed cluster:

        kubectl edit kaascephcluster -n <managedClusterProjectName>
        

        Substitute <managedClusterProjectName> with a corresponding value.

  2. In the clusterNet and/or publicNet parameters of the cephClusterSpec.network section, define a comma-separated array of CIDRs. For example:

    network:
      publicNet:  10.12.0.0/24,10.13.0.0/24
      clusterNet: 10.10.0.0/24,10.11.0.0/24
    
  3. Select from the following options:

    • If you are bootstrapping a management cluster, save the updated KaaSCephCluster template to the templates/bm/kaascephcluster.yaml.template file and proceed with the bootstrap.

    • If you are creating a managed cluster, save the updated KaaSCephCluster template to the corresponding file and proceed with the managed cluster creation.

    • If you are configuring KaaSCephCluster of an existing management cluster, run the following command:

      kubectl apply
      
    • If you are configuring KaaSCephCluster of an existing managed cluster, run the following command:

      kubectl apply -n <managedClusterProjectName>
      

      Substitute <managedClusterProjectName> with the corresponding value.

Once done, the specified network CIDRs will be passed to the Ceph daemons pods through the rook-config-override ConfigMap.