Expand IP addresses capacity in an existing cluster

If the subnet capacity on your existing cluster is not enough to add new machines, use the l2TemplateSelector feature to expand the IP addresses capacity:

  1. Create new Subnet object(s) to define additional address ranges for new machines.

  2. Set up routing between the existing and new subnets.

  3. Create new L2 template(s) with the new subnet(s) being used in l3Layout.

  4. Set up l2TemplateSelector in the Machine objects for new machines.

To expand IP addresses capacity for an existing cluster:

  1. Verify the capacity of the subnet(s) currently associated with the L2 template(s) used for cluster deployment:

    • If labelSelector is not used for the given subnet, use the namespace value of the L2 template and the subnetName value from the l3Layout section:

      kubectl get subnet -n <namespace> <subnetName>
      
    • If labelSelector is used for the given subnet, use the namespace value of the L2 template and comma-separated key-value pairs from the labelSelector section for the given subnet in the l3Layout section:

      kubectl get subnet -n <namespace> -l <key1=value1>[<,key2=value2>...]
      

      Example command:

      kubectl get subnet -n test-ns -l cluster.sigs.k8s.io/cluster-name=managed123,user-defined/purpose=lcm-base
      

      Example of system response:

      NAME             AGE  CIDR            GATEWAY      CAPACITY  ALLOCATABLE  STATUS
      old-lcm-network  8d   192.168.1.0/24  192.168.1.1  253       0            OK
      
    Existing Subnet example
     apiVersion: "ipam.mirantis.com/v1alpha1"
     kind: Subnet
     metadata:
       name: old-lcm-network
       namespace: test-ns
       labels:
         cluster.sigs.k8s.io/cluster-name: managed123
         ipam/SVC-k8s-lcm: "1"
         kaas.mirantis.com/provider: baremetal
         kaas.mirantis.com/region: region-one
         user-defined/purpose: lcm-base
     spec:
       cidr: 192.168.1.0/24
       gateway: 192.168.1.1
       . . .
     status:
       allocatable: 0
       allocatedIPs:
         . . .
       capacity: 253
       cidr: 192.168.1.0/24
       gateway: 192.168.1.1
       ranges:
         - 192.168.1.2-192.168.1.254
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

    Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

    Existing L2 template example
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: L2Template
    metadata:
      name: base-template
      namespace: test-ns
      labels:
        ipam/DefaultForCluster: "1"
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        cluster.sigs.k8s.io/cluster-name: managed123
    spec:
      autoIfMappingPrio:
        - provision
        - eno
        - ens
      l3Layout:
        - scope: namespace
          subnetName: lcm-subnet1
          labelSelector:
            cluster.sigs.k8s.io/cluster-name: managed123
            user-defined/purpose: lcm-base
      npTemplate: |
        version: 2
        renderer: networkd
        ethernets:
          {{nic 0}}:
            match:
              macaddress: {{mac 0}}
            set-name: {{nic 0}}
        bridges:
          k8s-lcm:
            interfaces:
              - {{nic 0}}
            addresses:
              - {{ip "k8s-lcm:lcm-subnet1"}}
            gateway4: {{gateway_from_subnet "lcm-subnet1"}}
            nameservers:
              addresses: {{nameservers_from_subnet "lcm-subnet1"}}
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

    Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

    Note

    Before MOSK 23.3, an L2 template requires clusterRef: <clusterName> in the spec section. Since MOSK 23.3, this parameter is deprecated and automatically migrated to the cluster.sigs.k8s.io/cluster-name: <clusterName> label.

  2. Create new objects:

    • Subnet with the user-defined/purpose: lcm-additional label.

    • L2Template with the alternative-template: “1” label. The L2 template should reference the new Subnet object using the user-defined/purpose: lcm-additional label in the labelSelector field.

    Note

    The label name user-defined/purpose is used for illustration purposes. Use any custom label name that differs from system names. Use of a unique prefix such as user-defined/ is recommended.

    New subnet example
    apiVersion: "ipam.mirantis.com/v1alpha1"
    kind: Subnet
    metadata:
      name: new-lcm-network
      namespace: test-ns
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        ipam/SVC-k8s-lcm: "1"
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        user-defined/purpose: lcm-additional
    spec:
      cidr: 192.168.200.0/24
      gateway: 192.168.200.1
      . . .
    status:
      allocatable: 253
      allocatedIPs:
        . . .
      capacity: 253
      cidr: 192.168.200.0/24
      gateway: 192.168.200.1
      ranges:
        - 192.168.200.2-192.168.200.254
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

    Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

    Alternative L2 template example
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: L2Template
    metadata:
      name: alternative-template
      namespace: test-ns
      labels:
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
        cluster.sigs.k8s.io/cluster-name: managed123
        alternative-template: “1”
    spec:
      autoIfMappingPrio:
        - provision
        - eno
        - ens
      l3Layout:
        - scope: namespace
          subnetName: lcm-subnet2
          labelSelector:
            cluster.sigs.k8s.io/cluster-name: managed123
            user-defined/purpose: lcm-additional
      npTemplate: |
        version: 2
        renderer: networkd
        ethernets:
          {{nic 0}}:
            match:
              macaddress: {{mac 0}}
            set-name: {{nic 0}}
        bridges:
          k8s-lcm:
            interfaces:
              - {{nic 0}}
            addresses:
              - {{ip "k8s-lcm:lcm-subnet2"}}
            gateway4: {{gateway_from_subnet "lcm-subnet2"}}
            nameservers:
              addresses: {{nameservers_from_subnet "lcm-subnet2"}}
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

    Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

    Note

    Before MOSK 23.3, an L2 template requires clusterRef: <clusterName> in the spec section. Since MOSK 23.3, this parameter is deprecated and automatically migrated to the cluster.sigs.k8s.io/cluster-name: <clusterName> label.

    You can also reference the new Subnet object by using its name in the l3Layout section of the alternative-template L2 template.

    Snippet example of an alternative L2 template
    ...
    spec:
      ...
      l3Layout:
        - scope: namespace
          subnetName: new-lcm-network
      ...
      npTemplate: |
        ...
        bridges:
          k8s-lcm:
            interfaces:
              - {{nic 0}}
            addresses:
              - {{ip "k8s-lcm:new-lcm-network"}}
            gateway4: {{gateway_from_subnet "new-lcm-network"}}
            nameservers:
              addresses: {{nameservers_from_subnet "new-lcm-network"}}
    
  3. Set up IP routing between the existing and new subnets using the tools of your cloud network infrastructure.

  4. In the providerSpec section of the new Machine object, define the alternative-template label for l2TemplateSelector:

    Snippet example of the new Machine object
    apiVersion: cluster.k8s.io/v1alpha1
    kind: Machine
    metadata:
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        kaas.mirantis.com/provider: baremetal
        kaas.mirantis.com/region: region-one
      name: additional-machine
      namespace: test-ns
    spec:
      ...
      providerSpec:
        value:
          ...
          l2TemplateSelector:
            label: alternative-template
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

    Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

    After creation, the new machine will use the alternative L2 template that uses the new-lcm-network subnet linked by L3Layout.

  5. Optional. Configure an additional IP address pool for MetalLB:

    Configure the additional extension IP address pool for the metallb load balancer service.

    1. Open the MetalLBConfig object of the management cluster for editing:

      kubectl edit metallbconfig <MetalLBConfigName>
      
    2. In the ipAddressPools section, add:

      ...
      spec:
        ipAddressPools:
        - name: extension
          spec:
            addresses:
            - <pool_start_ip>-<pool_end_ip>
            autoAssign: false
            avoidBuggyIPs: false
      ...
      

      In the snippet above, replace the following parameters:

      • <pool_start_ip> - first IP address in the required range

      • <pool_end_ip> - last IP address in the range

    3. Add the extension IP address pool name to the L2Advertisements definition. You can add it to the same L2 advertisement as the default IP address pool, or create a new L2 advertisement if required.

      ...
      spec:
        l2Advertisements:
        - name: default
          spec:
            interfaces:
            - k8s-lcm
            ipAddressPools:
            - default
            - extension
      ...
      
    4. Save and exit the object to apply changes.

    Define additional address ranges for MetalLB. For details, see the optional step for the MetalLB service in create-subnet-cli.

    You can create one or several Subnet objects to extend the MetalLB address pool with additional ranges. When the MetalLB traffic is routed through the default gateway, you can add the MetalLB address ranges that belong to different CIDR subnet addresses.

    For example:

    MetalLB configuration example snippet
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: Subnet
    metadata:
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        ipam/SVC-MetalLB: '1'
        kaas.mirantis.com/region: region-one
        kaas.mirantis.com/provider: baremetal
        user-defined/purpose: metallb-additional1
      name: metallb-svc1-for-managed
      namespace: test-ns
    spec:
      cidr: 172.16.168.0/24
      includeRanges:
      - 172.16.168.11-172.16.168.30
      - 172.16.168.41-172.16.168.50
    
    apiVersion: ipam.mirantis.com/v1alpha1
    kind: Subnet
    metadata:
      labels:
        cluster.sigs.k8s.io/cluster-name: managed123
        ipam/SVC-MetalLB: '1'
        kaas.mirantis.com/region: region-one
        kaas.mirantis.com/provider: baremetal
        user-defined/purpose: metallb-additional2
      name: metallb-svc2-for-managed
      namespace: test-ns
    spec:
      cidr: 172.16.169.0/24
      includeRanges:
      - 172.16.169.11-172.16.169.20
    

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

    Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

  6. Verify the created objects for MetalLB. For reference, use the following files in Container Cloud documentation: Example of a complete L2 templates configuration for cluster creation:

    • Since MOSK 24.2: managed-ns_MetalLBConfig-lb-managed.yaml

    • Before MOSK 24.2: managed-ns_Subnet_metallb-public-for-managed.yaml