Configure MetalLB

This section describes how to set up and verify MetalLB parameters during configuration of subnets for a MOSK cluster creation.

To configure MetalLB:

  1. Optional. Configure parameters related to MetalLB components life cycle such as deployment and update using the metallb Helm chart values in the Cluster spec section. For example:

  2. Configure the MetalLB parameters related to LB address allocation and announcement for cluster services:

    Select from the following options:

    • Recommended. Default. Technology Preview. Create MetalLBConfig and MetalLBConfigTemplate objects. This method allows using the Subnet object to define MetalLB address pools. For details, see Container Cloud API documentation: MetalLBConfig resource and MetalLBConfigTemplate resource.

      For configuration examples that use MetalLBConfig and MetalLBConfigTemplate, see the following sections in Container Cloud documentation:

      Since MOSK 23.2.2, in the Technology Preview scope, you can use BGP for announcement of external addresses of Kubernetes load-balanced services for a MOSK cluster. To configure the BGP announcement mode for MetalLB, use MetalLBConfig and MetalLBConfigTemplate objects.

      The use of BGP is required to announce services LB IP addresses when using MetalLB on nodes that are distributed across multiple racks. In this case, setting of rack-id labels on nodes is required, they are used in node selectors for BGPPeer, BGPAdvertisement, or both MetalLB objects to properly configure BGP connections from each node.

      Configuration example of the Machine object for the BGP announcement mode
      apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-compute-1
  namespace: mosk-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    ipam/RackRef: rack-1  # reference to the "rack-1" Rack
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  providerSpec:
    value:
      ...
      nodeLabels:
      - key: rack-id   # node label can be used in "nodeSelectors" inside
        value: rack-1  # "BGPPeer" and/or "BGPAdvertisement" MetalLB objects
  ...
      Configuration example of the MetalLBConfigTemplate object for the BGP announcement mode
      apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfigTemplate
metadata:
  name: test-cluster-metallb-config-template
  namespace: mosk-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
spec:
  templates:
    ...
    bgpPeers: |
      - name: svc-peer-1
        spec:
          peerAddress: 10.77.42.1
          peerASN: 65100
          myASN: 65101
          nodeSelectors:
            - matchLabels:
                rack-id: rack-1  # references the nodes having
                                 # the "rack-id=rack-1" label
    bgpAdvertisements: |
      - name: services
        spec:
          ipAddressPools:
            - services
          peers:
            - svc-peer-1
            ...

      The bgpPeers and bgpAdvertisements fields are used to configure BGP announcement instead of l2Advertisements.

      The use of BGP for announcement also allows for better balancing of service traffic between cluster nodes as well as gives more configuration control and flexibility for infrastructure administrators. For configuration examples, refer to Examples of MetalLBConfigTemplate. For configuration procedure, refer to Configure BGP announcement for cluster API LB address.

      The following rules and requirements apply to configuration of MetalLBConfig and MetalLBConfigTemplate:

      • Define one MetalLBConfig and MetalLBConfigTemplate object per cluster.

      • Define the following mandatory labels:

        cluster.sigs.k8s.io/cluster-name

        Specifies the cluster name where the MetalLB address pool is used.

        kaas.mirantis.com/region

        Specifies the region name of the cluster where the MetalLB address pool is used.

        kaas.mirantis.com/provider

        Specifies the provider of the cluster where the MetalLB address pool is used.

      • You can use MetalLBConfig without MetalLBConfigTemplate but not the opposite way.

      • You can use the MetalLBConfig and MetalLBConfigTemplate objects to optimize the announcement of the services LB addresses using the interfaces selector for the l2Advertisements object. This selector allows for announcement of addresses only on selected host interfaces. For details, see Container Cloud API Reference: MetalLBConfigTemplate spec.

        Mirantis recommends this configuration if nodes use separate host networks for different types of traffic. The pros of such configuration are as follows: less spam on other interfaces and networks, limited chances to reach services LB addresses from irrelevant interfaces and networks.

      • When using MetalLBConfigTemplate:

        • MetalLBConfig must reference MetalLBConfigTemplate by name:

          spec:
  templateName: <metallb-template-name>

        • You can use Subnet objects for defining MetalLB address pools. Refer to MetalLB configuration guidelines for subnets for guidelines on configuring MetalLB address pools using Subnet objects.

        The use of MetalLBConfigTemplate object gives more flexibility when describing MetalLB configuration as it allows using variables, functions, and the Go template expressions inside object templates.

      • Intersection of IP address ranges within any single MetalLB address pool is not allowed.

      • At least one MetalLB address pool must have the auto-assign policy enabled so that unannotated services can have load balancer IPs allocated for them.

      • When configuring multiple address pools with the auto-assign policy enabled, keep in mind that it is not determined in advance which pool of those multiple address pools is used to allocate an IP for a particular unannotated service.

    • Deprecated since MOSK 23.2. Configure the configInline value in the MetalLB chart of the Cluster object. This functionality will be removed in one of the following releases.

    • Deprecated since MOSK 23.2. Configure the Subnet objects without MetalLBConfigTemplate. This functionality will be removed in one of the following releases.

    Caution

    If the MetalLBConfig object is not used for MetalLB configuration related to address allocation and announcement of services LB, then automated migration applies cluster creation of update to MOSK 23.2.

    During automated migration, the MetalLBConfig and MetalLBConfigTemplate objects are created and contents of the MetalLB chart configInline value is converted to the parameters of the MetalLBConfigTemplate object.

    Any change to the configInline value made on a MOSK 23.2 cluster will be reflected in the MetalLBConfigTemplate object.

    This automated migration will be removed in one of the following releases together with the possibility to use the configInline value of the MetalLB chart. After that, any changes in MetalLB configuration related to address allocation and announcement of services LB will be applied using the MetalLBConfigTemplate and Subnet objects only.

    Select from the following options:

    • Configure the configInline value for the MetalLB chart in the Cluster object.

      Caution

      The use of configInline value for the MetalLB chart in the Cluster object is deprecated since MOSK 23.2 and will be removed in one of the following releases.

    • Configure Subnet objects. For details, see MetalLB configuration guidelines for subnets.

    • Configure both the configInline value for the MetalLB chart and Subnet objects.

      The resulting MetalLB address pools configuration will contain address ranges from both cluster specification and Subnet objects. All address ranges for L2 address pools will be aggregated into a single L2 address pool and sorted as strings.

      Caution

      The use of configInline value for the MetalLB chart in the Cluster object is deprecated since MOSK 23.2 and will be removed in one of the following releases.

    • Technology Preview since MOSK 22.5. Configure MetalLB address pools and advertisement parameters using the MetalLBConfig object. If you plan to use the MetalLBConfig object in future, refer to the configuration steps for MOSK 23.2 or later.

      You can use this object instead of Subnet objects and MetalLB chart values in the Cluster object. Do not mix both configuration methods.

      1. Verify that no subnets with the ipam/SVC-MetalLB label are defined for a particular cluster.

      2. Verify that the configInline value is absent for the MetalLB chart in the Cluster object. Other fields in values for the MetalLB chart are allowed.

      3. Create the MetalLBConfig object that must reference the target cluster name. Define one MetalLBConfig object per cluster.

      4. Add MetalLB configuration parameters to the MetalLBConfig object definition as described in Container Cloud API documentation: MetalLBConfig resource.

  3. Verify the current MetalLB configuration:

    Verify the MetalLB configuration that is stored in MetalLB objects:

    kubectl -n metallb-system get ipaddresspools,l2advertisements

    The example system output:

    NAME                                    AGE
ipaddresspool.metallb.io/default        129m
ipaddresspool.metallb.io/services-pxe   129m

NAME                                      AGE
l2advertisement.metallb.io/default        129m
l2advertisement.metallb.io/services-pxe   129m

    Verify one of the listed above MetalLB objects:

    kubectl -n metallb-system get <object> -o json | jq '.spec'

    The example system output for ipaddresspool objects:

    $ kubectl -n metallb-system get ipaddresspool.metallb.io/default -o json | jq '.spec'
{
  "addresses": [
    "10.0.11.61-10.0.11.80"
  ],
  "autoAssign": true,
  "avoidBuggyIPs": false
}
$ kubectl -n metallb-system get ipaddresspool.metallb.io/services-pxe -o json | jq '.spec'
{
  "addresses": [
    "10.0.0.61-10.0.0.70"
  ],
  "autoAssign": false,
  "avoidBuggyIPs": false
}

    Verify the MetalLB configuration that is stored in the ConfigMap object:

    kubectl -n metallb-system get cm metallb -o jsonpath={.data.config}

    An example of a successful output:

    address-pools:
- name: default
  protocol: layer2
  addresses:
  - 10.0.11.61-10.0.11.80
- name: services-pxe
  protocol: layer2
  auto-assign: false
  addresses:
  - 10.0.0.61-10.0.0.70

    The auto-assign parameter will be set to false for all address pools except the default one. So, a particular service will get an address from such an address pool only if the Service object has a special metallb.universe.tf/address-pool annotation that points to the specific address pool name.