Expand IP addresses capacity in an existing cluster

If the subnet capacity on your existing cluster is not enough to add new machines, use the l2TemplateSelector feature to expand the IP addresses capacity:

  1. Create new Subnet object(s) to define additional address ranges for new machines.

  2. Set up routing between the existing and new subnets.

  3. Create new L2 template(s) with the new subnet(s) being used in l3Layout.

  4. Set up l2TemplateSelector in the Machine objects for new machines.

To expand IP addresses capacity for an existing cluster:

  1. Verify the capacity of the subnet(s) currently associated with the L2 template(s) used for cluster deployment:

    • If labelSelector is not used for the given subnet, use the namespace value of the L2 template and the subnetName value from the l3Layout section:

      kubectl get subnet -n <namespace> <subnetName>
      
    • If labelSelector is used for the given subnet, use the namespace value of the L2 template and comma-separated key-value pairs from the labelSelector section for the given subnet in the l3Layout section:

      kubectl get subnet -n <namespace> -l <key1=value1>[<,key2=value2>...]
      

      Example command:

      kubectl get subnet -n test-ns -l cluster.sigs.k8s.io/cluster-name=managed123,user-defined/purpose=lcm-base
      

      Example of system response:

      NAME             AGE  CIDR            GATEWAY      CAPACITY  ALLOCATABLE  STATUS
      old-lcm-network  8d   192.168.1.0/24  192.168.1.1  253       0            OK
      
    Existing Subnet example
     apiVersion: "ipam.mirantis.com/v1alpha1"
     kind: Subnet
     metadata:
       name: old-lcm-network
       namespace: test-ns
       labels:
         cluster.sigs.k8s.io/cluster-name: managed123
         ipam/SVC-k8s-lcm: "1"
         kaas.mirantis.com/provider: baremetal
         kaas.mirantis.com/region: region-one
         user-defined/purpose: lcm-base
     spec:
       cidr: 192.168.1.0/24
       gateway: 192.168.1.1
       . . .
     status:
       allocatable: 0
       allocatedIPs:
         . . .
       capacity: 253
       cidr: 192.168.1.0/24
       gateway: 192.168.1.1
       ranges:
         - 192.168.1.2-192.168.1.254
       state: OK
    
    Existing L2 template example
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: L2Template
    metadata:
      name: base-template
      namespace: test-ns
      labels:
        ipam/DefaultForCluster: "1"
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        cluster.sigs.k8s.io/cluster-name: managed123
    spec:
      autoIfMappingPrio:
        - provision
        - eno
        - ens
      l3Layout:
        - scope: namespace
          subnetName: lcm-subnet1
          labelSelector:
            cluster.sigs.k8s.io/cluster-name: managed123
            user-defined/purpose: lcm-base
      npTemplate: |
        version: 2
        renderer: networkd
        ethernets:
          {{nic 0}}:
            match:
              macaddress: {{mac 0}}
            set-name: {{nic 0}}
        bridges:
          k8s-lcm:
            interfaces:
              - {{nic 0}}
            addresses:
              - {{ip "k8s-lcm:lcm-subnet1"}}
            gateway4: {{gateway_from_subnet "lcm-subnet1"}}
            nameservers:
              addresses: {{nameservers_from_subnet "lcm-subnet1"}}
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.

  2. Create new objects:

    • Subnet with the user-defined/purpose: lcm-additional label.

    • L2Template with the alternative-template: “1” label. The L2 template should reference the new Subnet object using the user-defined/purpose: lcm-additional label in the labelSelector field.

    Note

    The label name user-defined/purpose is used for illustration purposes. Use any custom label name that differs from system names. Use of a unique prefix such as user-defined/ is recommended.

    New subnet example
    apiVersion: "ipam.mirantis.com/v1alpha1"
    kind: Subnet
    metadata:
      name: new-lcm-network
      namespace: test-ns
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        ipam/SVC-k8s-lcm: "1"
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        user-defined/purpose: lcm-additional
    spec:
      cidr: 192.168.200.0/24
      gateway: 192.168.200.1
      . . .
    status:
      allocatable: 253
      allocatedIPs:
        . . .
      capacity: 253
      cidr: 192.168.200.0/24
      gateway: 192.168.200.1
      ranges:
        - 192.168.200.2-192.168.200.254
      state: OK
    
    Alternative L2 template example
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: L2Template
    metadata:
      name: alternative-template
      namespace: test-ns
      labels:
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        cluster.sigs.k8s.io/cluster-name: managed123
        alternative-template: “1”
    spec:
      autoIfMappingPrio:
        - provision
        - eno
        - ens
      l3Layout:
        - scope: namespace
          subnetName: lcm-subnet2
          labelSelector:
            cluster.sigs.k8s.io/cluster-name: managed123
            user-defined/purpose: lcm-additional
      npTemplate: |
        version: 2
        renderer: networkd
        ethernets:
          {{nic 0}}:
            match:
              macaddress: {{mac 0}}
            set-name: {{nic 0}}
        bridges:
          k8s-lcm:
            interfaces:
              - {{nic 0}}
            addresses:
              - {{ip "k8s-lcm:lcm-subnet2"}}
            gateway4: {{gateway_from_subnet "lcm-subnet2"}}
            nameservers:
              addresses: {{nameservers_from_subnet "lcm-subnet2"}}
    

    You can also reference the new Subnet object by using its name in the l3Layout section of the alternative-template L2 template.

    Alternative L2 template example
    ...
    spec:
      ...
      l3Layout:
        - scope: namespace
          subnetName: new-lcm-network
      ...
      npTemplate: |
        ...
        bridges:
          k8s-lcm:
            interfaces:
              - {{nic 0}}
            addresses:
              - {{ip "k8s-lcm:new-lcm-network"}}
            gateway4: {{gateway_from_subnet "new-lcm-network"}}
            nameservers:
              addresses: {{nameservers_from_subnet "new-lcm-network"}}
    
  3. Set up IP routing between the existing and new subnets using the tools of your cloud network infrastructure.

  4. In the providerSpec section of the new Machine object, define the alternative-template label for l2TemplateSelector:

    Snippet example of the new Machine object
    apiVersion: cluster.k8s.io/v1alpha1
    kind: Machine
    metadata:
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
      name: additional-machine
      namespace: test-ns
    spec:
      ...
      providerSpec:
        value:
          ...
          l2TemplateSelector:
            label: alternative-template
    

    After creation, the new machine will use the alternative L2 template that uses the new-lcm-network subnet linked by L3Layout.

  5. Define additional address ranges for MetalLB. For details, see the optional step for the MetalLB service in Create subnets for a managed cluster using CLI.

    You can create one or several Subnet objects to extend the MetalLB address pool with additional ranges. When the MetalLB traffic is routed through the default gateway, you can add the MetalLB address ranges that belong to different CIDR subnet addresses.

    For example:

    MetalLB configuration example snippet
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: Subnet
    metadata:
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        ipam/SVC-MetalLB: '1'
        kaas.mirantis.com/region: region-one
        kaas.mirantis.com/provider: baremetal
        user-defined/purpose: metallb-additional1
      name: metallb-svc1-for-managed
      namespace: test-ns
    spec:
      cidr: 172.16.168.0/24
      includeRanges:
      - 172.16.168.11-172.16.168.30
      - 172.16.168.41-172.16.168.50
    
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: Subnet
    metadata:
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        ipam/SVC-MetalLB: '1'
        kaas.mirantis.com/region: region-one
        kaas.mirantis.com/provider: baremetal
        user-defined/purpose: metallb-additional2
      name: metallb-svc2-for-managed
      namespace: test-ns
    spec:
      cidr: 172.16.169.0/24
      includeRanges:
      - 172.16.169.11-172.16.169.20
    
  6. Verify the created objects for MetalLB.

    For reference, use managed-ns_Subnet_metallb-public-for-managed.yaml in Example of a complete L2 templates configuration for cluster creation.