Configure MetalLB

This section describes how to set up and verify MetalLB parameters during configuration of subnets for a managed cluster creation.

Caution

This section also applies to the bootstrap procedure of a management cluster with the following differences:

  • Instead of the Cluster object, configure templates/bm/cluster.yaml.template.

  • Instead of the MetalLBConfig object, configure templates/bm/metallbconfig.yaml.template.

  • Instead of creating specific IPAM objects such as Subnet, add their configuration to templates/bm/ipam-objects.yaml.template.

The Kubernetes objects described below are created for a management cluster from template files during bootstrap.

Configuration rules for ‘MetalLBConfig’ and ‘MetalLBConfigTemplate’ objects

Caution

The use of the MetalLBConfig object is mandatory for management and managed clusters after a management cluster upgrade to the Cluster release 17.0.0.

The following rules and requirements apply to configuration of the MetalLBConfig and MetalLBConfigTemplate objects:

  • Define one MetalLBConfig object per cluster.

  • Define the following mandatory labels:

    cluster.sigs.k8s.io/cluster-name

    Specifies the cluster name where the MetalLB address pool is used.

    kaas.mirantis.com/provider

    Specifies the provider of the cluster where the MetalLB address pool is used.

    kaas.mirantis.com/region

    Specifies the region name of the cluster where the MetalLB address pool is used.

    Note

    The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.

  • Intersection of IP address ranges within any single MetalLB address pool is not allowed.

  • At least one MetalLB address pool must have the auto-assign policy enabled so that unannotated services can have load balancer IP addresses allocated to them.

  • When configuring multiple address pools with the auto-assign policy enabled, keep in mind that it is not determined in advance which pool of those multiple address pools is used to allocate an IP address for a particular unannotated service.

  • You can use the MetalLBConfig object to optimize address announcement for load-balanced services using the interfaces selector for the l2Advertisements object. This selector allows for address announcement only on selected host interfaces. For details, see API Reference: MetalLB configuration examples.

    Note

    Before Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0), use the deprecated MetalLBConfigTemplate object along with MetalLBConfig for this purpose. For details, see API Reference: MetalLBConfigTemplate spec.

  • Optional. Deprecated since Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0) and will be removed in one of the following releases. Define one MetalLBConfigTemplate object per cluster. The use of this object without MetalLBConfig is not allowed.

  • When using MetalLBConfigTemplate:

    • MetalLBConfig must reference MetalLBConfigTemplate by name:

      spec:
  templateName: <managed-metallb-template>

    • You can use Subnet objects for defining MetalLB address pools. Refer to MetalLB configuration guidelines for subnets for guidelines on configuring MetalLB address pools using Subnet objects.

Configure and verify MetalLB

  1. Optional. Configure parameters related to MetalLB components life cycle such as deployment and update using the metallb Helm chart values in the Cluster spec section. For example:

  2. Configure the MetalLB parameters related to IP address allocation and announcement for load-balanced cluster services. Select from the following options:

    Recommended. Default. Mandatory after a management cluster upgrade to the Cluster release 17.2.0.

    Create the MetalLBConfig object:

    In the Technology Preview scope, you can use BGP for announcement of external addresses of Kubernetes load-balanced services for managed clusters. To configure the BGP announcement mode for MetalLB, use the MetalLBConfig object.

    The use of BGP is required to announce IP addresses for load-balanced services when using MetalLB on nodes that are distributed across multiple racks. In this case, setting of rack-id labels on nodes is required, they are used in node selectors for BGPPeer, BGPAdvertisement, or both MetalLB objects to properly configure BGP connections from each node.

    Configuration example of the Machine object for the BGP announcement mode
    apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-compute-1
  namespace: managed-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    ipam/RackRef: rack-1  # reference to the "rack-1" Rack
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  providerSpec:
    value:
      ...
      nodeLabels:
      - key: rack-id   # node label can be used in "nodeSelectors" inside
        value: rack-1  # "BGPPeer" and/or "BGPAdvertisement" MetalLB objects
  ...
    Configuration example of the MetalLBConfig object for the BGP announcement mode
    apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
  name: test-cluster-metallb-config
  namespace: managed-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  ...
  bgpPeers:
    - name: svc-peer-1
      spec:
        holdTime: 0s
        keepaliveTime: 0s
        peerAddress: 10.77.42.1
        peerASN: 65100
        myASN: 65101
        nodeSelectors:
          - matchLabels:
            rack-id: rack-1  # references the nodes having
                             # the "rack-id=rack-1" label
  bgpAdvertisements:
    - name: services
      spec:
        aggregationLength: 32
        aggregationLengthV6: 128
        ipAddressPools:
          - services
        peers:
          - svc-peer-1
          ...

    Select from the following options:

    • Deprecated since the Cluster releases 17.2.0 and 16.2.0. Mandatory after a management cluster upgrade to the Cluster release 17.0.0.

      Create MetalLBConfig and MetalLBConfigTemplate objects. This method allows using the Subnet object to define MetalLB address pools.

      Note

      For managed clusters, this configuration method is generally available since Cluster releases 17.0.0 and 16.0.0. And it is available as Technology Preview since Cluster releases 15.0.1, 14.0.1, and 14.0.0.

      Since Cluster releases 15.0.3 and 14.0.3, in the Technology Preview scope, you can use BGP for announcement of external addresses of Kubernetes load-balanced services for managed clusters. To configure the BGP announcement mode for MetalLB, use MetalLBConfig and MetalLBConfigTemplate objects.

      The use of BGP is required to announce IP addresses for load-balanced services when using MetalLB on nodes that are distributed across multiple racks. In this case, setting of rack-id labels on nodes is required, they are used in node selectors for BGPPeer, BGPAdvertisement, or both MetalLB objects to properly configure BGP connections from each node.

      Configuration example of the Machine object for the BGP announcement mode
      apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-compute-1
  namespace: managed-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    ipam/RackRef: rack-1  # reference to the "rack-1" Rack
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  providerSpec:
    value:
      ...
      nodeLabels:
      - key: rack-id   # node label can be used in "nodeSelectors" inside
        value: rack-1  # "BGPPeer" and/or "BGPAdvertisement" MetalLB objects
  ...
      Configuration example of the MetalLBConfigTemplate object for the BGP announcement mode
      apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfigTemplate
metadata:
  name: test-cluster-metallb-config-template
  namespace: managed-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  templates:
    ...
    bgpPeers: |
      - name: svc-peer-1
        spec:
          peerAddress: 10.77.42.1
          peerASN: 65100
          myASN: 65101
          nodeSelectors:
            - matchLabels:
                rack-id: rack-1  # references the nodes having
                                 # the "rack-id=rack-1" label
    bgpAdvertisements: |
      - name: services
        spec:
          ipAddressPools:
            - services
          peers:
            - svc-peer-1
            ...

      The bgpPeers and bgpAdvertisements fields are used to configure BGP announcement instead of l2Advertisements.

      Note

      The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.

      The use of BGP for announcement also allows for better balancing of service traffic between cluster nodes as well as gives more configuration control and flexibility for infrastructure administrators. For configuration examples, refer to MetalLB configuration examples. For configuration procedure, refer to Configure BGP announcement for cluster API LB address.

    • Deprecated since Container Cloud 2.24.0. Configure the configInline value in the MetalLB chart of the Cluster object.

      Warning

      This functionality is removed during the management cluster upgrade to the Cluster release 17.0.0. Therefore, this option becomes unavailable on managed clusters after the parent management cluster upgrade to 17.0.0.

    • Deprecated since Container Cloud 2.24.0. Configure the Subnet objects without MetalLBConfigTemplate.

      Warning

      This functionality is removed during the management cluster upgrade to the Cluster release 17.0.0. Therefore, this option becomes unavailable on managed clusters after the parent management cluster upgrade to 17.0.0.

    Caution

    If the MetalLBConfig object is not used for MetalLB configuration related to address allocation and announcement for load-balanced services, then automated migration applies during creation of clusters of any type or cluster update to Cluster releases 15.0.x or 14.0.x.

    During automated migration, the MetalLBConfig and MetalLBConfigTemplate objects are created and contents of the MetalLB chart configInline value is converted to the parameters of the MetalLBConfigTemplate object.

    Any change to the configInline value made on a 15.0.x or 14.0.x cluster will be reflected in the MetalLBConfigTemplate object.

    This automated migration is removed during your management cluster upgrade to the Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0, together with the possibility to use the configInline value of the MetalLB chart. After that, any changes in MetalLB configuration related to address allocation and announcement for load-balanced services will be applied using the MetalLBConfigTemplate and Subnet objects only.

    Select from the following options:

    • Configure Subnet objects. For details, see MetalLB configuration guidelines for subnets.

    • Configure the configInline value for the MetalLB chart in the Cluster object.

    • Configure both the configInline value for the MetalLB chart and Subnet objects.

      The resulting MetalLB address pools configuration will contain address ranges from both cluster specification and Subnet objects. All address ranges for L2 address pools will be aggregated into a single L2 address pool and sorted as strings.

    Changes to be applied since Container Cloud 2.25.0

    The configuration options above are deprecated since Container Cloud 2.24.0, after your management cluster upgrade to the Cluster release 14.0.0 or 14.0.1. Automated migration of MetalLB parameters applies during cluster creation or update to Container Cloud 2.24.x.

    During automated migration, the MetalLBConfig and MetalLBConfigTemplate objects are created and contents of the MetalLB chart configInline value is converted to the parameters of the MetalLBConfigTemplate object.

    Any change to the configInline value made on a Container Cloud 2.24.x cluster will be reflected in the MetalLBConfigTemplate object.

    This automated migration is removed during your management cluster upgrade to the Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0, together with the possibility to use the configInline value of the MetalLB chart. After that, any changes in MetalLB configuration related to address allocation and announcement for load-balanced services will be applied using the MetalLBConfigTemplate and Subnet objects only.

  3. Verify the current MetalLB configuration:

    Verify the MetalLB configuration that is stored in MetalLB objects:

    kubectl -n metallb-system get ipaddresspools,l2advertisements

    The example system output:

    NAME                                    AGE
ipaddresspool.metallb.io/default        129m
ipaddresspool.metallb.io/services-pxe   129m

NAME                                      AGE
l2advertisement.metallb.io/default        129m
l2advertisement.metallb.io/services-pxe   129m

    Verify one of the listed above MetalLB objects:

    kubectl -n metallb-system get <object> -o json | jq '.spec'

    The example system output for ipaddresspool objects:

    $ kubectl -n metallb-system get ipaddresspool.metallb.io/default -o json | jq '.spec'
{
  "addresses": [
    "10.0.11.61-10.0.11.80"
  ],
  "autoAssign": true,
  "avoidBuggyIPs": false
}
$ kubectl -n metallb-system get ipaddresspool.metallb.io/services-pxe -o json | jq '.spec'
{
  "addresses": [
    "10.0.0.61-10.0.0.70"
  ],
  "autoAssign": false,
  "avoidBuggyIPs": false
}

    Verify the MetalLB configuration that is stored in the ConfigMap object:

    kubectl -n metallb-system get cm metallb -o jsonpath={.data.config}

    An example of a successful output:

    address-pools:
- name: default
  protocol: layer2
  addresses:
  - 10.0.11.61-10.0.11.80
- name: services-pxe
  protocol: layer2
  auto-assign: false
  addresses:
  - 10.0.0.61-10.0.0.70

    The auto-assign parameter will be set to false for all address pools except the default one. So, a particular service will get an address from such an address pool only if the Service object has a special metallb.universe.tf/address-pool annotation that points to the specific address pool name.

    Note

    It is expected that every Container Cloud service on a management cluster will be assigned to one of the address pools. Current consideration is to have two MetalLB address pools:

    • services-pxe is a reserved address pool name to use for the Container Cloud services in the PXE network (Ironic API, HTTP server, caching server).

    • default is an address pool to use for all other Container Cloud services in the management network. No annotation is required on the Service objects in this case.