Configure MetalLB¶

This section describes how to set up and verify MetalLB parameters before configuring subnets for a MOSK cluster.

Configuration rules for ‘MetalLBConfig’ and ‘MetalLBConfigTemplate’ objects¶

MetalLBConfigTemplate is deprecated since MOSK 24.2

Caution

The use of the MetalLBConfig object is mandatory after your management cluster upgrade to the Cluster release 17.0.0.

The following rules and requirements apply to configuration of the MetalLBConfig and MetalLBConfigTemplate objects:

Define one MetalLBConfig object per cluster.
Define the following mandatory labels:

cluster.sigs.k8s.io/cluster-name
Specifies the cluster name where the MetalLB address pool is used.

kaas.mirantis.com/region
Specifies the region name of the cluster where the MetalLB address pool is used.

kaas.mirantis.com/provider
Specifies the provider of the cluster where the MetalLB address pool is used.

Note

The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.
Intersection of IP address ranges within any single MetalLB address pool is not allowed.
At least one MetalLB address pool must have the auto-assign policy enabled so that unannotated services can have load balancer IP addresses allocated for them.
When configuring multiple address pools with the auto-assign policy enabled, keep in mind that it is not determined in advance which pool of those multiple address pools is used to allocate an IP address for a particular unannotated service.
You can use the MetalLBConfig object to optimize address announcement for load-balanced services using the interfaces selector for the l2Advertisements object. This selector allows for announcing addresses only on selected host interfaces. For details, see Container Cloud API Reference: MetalLBConfig spec.

Note

Before MOSK 24.2, use the deprecated MetalLBConfigTemplate object along with MetalLBConfig for this purpose. For details, see Container Cloud API Reference: MetalLBConfigTemplate spec.
Optional. Deprecated since MOSK 24.2 and will be removed in one of the following releases. Define one MetalLBConfigTemplate object per cluster. The use of this object without MetalLBConfig is not allowed.
When using MetalLBConfigTemplate:
- MetalLBConfig must reference MetalLBConfigTemplate by name:
```
spec:
  templateName: <managed-metallb-template>
```
- You can use Subnet objects for defining MetalLB address pools. Refer to MetalLB configuration guidelines for subnets for guidelines on configuring MetalLB address pools using Subnet objects.

Configure and verify MetalLB using the web UI¶

Available since MOSK 24.3

Note

The BGP configuration is not yet supported in the Container Cloud web UI. Meantime, use the CLI for this purpose. For details, see Configure and verify MetalLB using the CLI.

Optional. Configure parameters related to MetalLB components life cycle such as deployment and update using the metallb Helm chart values in the Cluster spec section. For example:
- Increase Pod resource limits for MetalLB as described in Limits for common components of any cluster type.
- Configure Pod node selectors or affinity for MetalLB speakers as described in Configure the MetalLB speaker node selector.
Log in to the Container Cloud web UI with the writer permissions.
Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.
In the Networks section, click the MetalLB Configs tab.
Click Create MetalLB Config.
Fill out the Create MetalLB Config form as required:
- Name
  Name of the MetalLB object being created.
- Cluster
  Name of the cluster that the MetalLB object is being created for
- IP Address Pools
  List of MetalLB IP address pool descriptions that will be used to create the MetalLB IPAddressPool objects. Click the + button on the right side of the section to add more objects.
  - Name
    IP address pool name.
  - Addresses
    Comma-separated ranges of the IP addresses included into the address pool.
  - Auto Assign
    Enable auto-assign policy for unannotated services to have load balancer IP addresses allocated to them. At least one MetalLB address pool must have the auto-assign policy enabled.
  - Service Allocation
    IP address pool allocation to services. Click Edit to insert a service allocation object with required label selectors for services in the YAML format. For example:
    
    serviceSelectors: - matchExpressions: - key: app.kubernetes.io/name operator: NotIn values: - dhcp-lb
    
    For details on the MetalLB IPAddressPool object type, see MetalLB documentation.
  - L2 Advertisements
    List of MetalLBL2Advertisement objects to create MetalLB L2Advertisement objects.
    
    The l2Advertisements object allows defining interfaces to optimize the announcement. When you use the interfaces selector, LB addresses are announced only on selected host interfaces.
    
    Mirantis recommends using the interfaces selector if nodes use separate host networks for different types of traffic. The pros of such configuration are as follows: less spam on other interfaces and networks and limited chances to reach IP addresses of load-balanced services from irrelevant interfaces and networks.
    
    Caution
    
    Interface names in the interfaces list must match those on the corresponding nodes.
    
    Add the following parameters:
    
    Name
    Name of the l2Advertisements object.
    
    Interfaces
    Optional. Comma-separated list of interface names that must match the ones on the corresponding nodes. These names are defined in L2 templates that are linked to the selected cluster.
    
    IP Address Pools
    Select the IP adress pool to use for the l2Advertisements object.
    
    Node Selectors
    Optional. Match labels and values for the Kubernetes node selector to limit the nodes announced as next hops for the LoadBalancer IP. If you do not provide any labels, all nodes are announced as next hops.
    
    For details on the MetalLB L2Advertisements object type, see MetalLB documentation.
Click Create.
In Networks > MetalLB Configs, verify the status of the created MetalLB object:
- Ready - object is operational.
- Error - object is non-operational. Hover over the status
  to obtain details of the issue.
Note

To verify the object details, in Networks > MetalLB Configs, click the More action icon in the last column of the required object section and select MetalLB Config info.
Proceed to creating cluster subnets as described in Create subnets.

Configure and verify MetalLB using the CLI¶

Optional. Configure parameters related to MetalLB components life cycle such as deployment and update using the metallb Helm chart values in the Cluster spec section. For example:
- Increase Pod resource limits for MetalLB as described in Limits for common components of any cluster type.
- Configure Pod node selectors or affinity for MetalLB speakers as described in Configure the MetalLB speaker node selector.
Configure the MetalLB parameters related to IP address allocation and announcement for load-balanced cluster services:
Since MOSK 24.2
Mandatory after a management cluster upgrade to the Cluster release 17.2.0. Recommended and default since MOSK 24.2.

Create the MetalLBConfig object:
- For configuration rules and requirements, see Configuration rules for ‘MetalLBConfig’ and ‘MetalLBConfigTemplate’ objects.
- For the object description, see Container Cloud API documentation: MetalLBConfig resource.
- For configuration examples that use MetalLBConfig, see the following sections in the Container Cloud documentation:
  - Example of a complete L2 templates configuration for cluster creation (the MetalLB configuration objects step)
  - Examples of MetalLBConfig
In the Technology Preview scope, you can use BGP for announcement of external addresses of Kubernetes load-balanced services for a MOSK cluster. To configure the BGP announcement mode for MetalLB, use MetalLBConfig object.

The use of BGP is required to announce IP addresses for load-balanced services when using MetalLB on nodes that are distributed across multiple racks. In this case, setting of rack-id labels on nodes is required, they are used in node selectors for BGPPeer, BGPAdvertisement, or both MetalLB objects to properly configure BGP connections from each node.
Configuration example of the Machine object for the BGP announcement mode
apiVersion: cluster.k8s.io/v1alpha1 kind: Machine metadata: name: test-cluster-compute-1 namespace: mosk-ns labels: cluster.sigs.k8s.io/cluster-name: test-cluster ipam/RackRef: rack-1 # reference to the "rack-1" Rack kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one spec: providerSpec: value: ... nodeLabels: - key: rack-id # node label can be used in "nodeSelectors" inside value: rack-1 # "BGPPeer" and/or "BGPAdvertisement" MetalLB objects ...

Note

The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.
Configuration example of the MetalLBConfig object for the BGP announcement mode
apiVersion: ipam.mirantis.com/v1alpha1 kind: MetalLBConfig metadata: name: test-cluster-metallb-config namespace: mosk-ns labels: cluster.sigs.k8s.io/cluster-name: test-cluster kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one spec: ... bgpPeers: - name: svc-peer-1 spec: holdTime: 0s keepaliveTime: 0s peerAddress: 10.77.42.1 peerASN: 65100 myASN: 65101 nodeSelectors: - matchLabels: rack-id: rack-1 # references the nodes having # the "rack-id=rack-1" label bgpAdvertisements: - name: services spec: aggregationLength: 32 aggregationLengthV6: 128 ipAddressPools: - services peers: - svc-peer-1 ...

Note

The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.

Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.

The bgpPeers and bgpAdvertisements fields are used to configure BGP announcement instead of l2Advertisements.
The use of BGP for announcement also allows for better balancing of service traffic between cluster nodes as well as gives more configuration control and flexibility for infrastructure administrators. For configuration examples, refer to Examples of MetalLBConfig. For configuration procedure, refer to Configure BGP announcement for cluster API LB address.
Since MOSK 23.2
Select from the following options:
- Deprecated since MOSK 24.2. Mandatory after a management cluster upgrade to the Cluster release 17.0.0. Recommended and default since MOSK 23.2 in the Technology Preview scope. Create MetalLBConfig and MetalLBConfigTemplate objects. This method allows using the Subnet object to define MetalLB address pools.
  - For configuration rules and requirements, see Configuration rules for ‘MetalLBConfig’ and ‘MetalLBConfigTemplate’ objects.
  - For objects description, see Container Cloud API documentation: MetalLBConfig resource and MetalLBConfigTemplate resource.
  - For configuration examples that use MetalLBConfig and MetalLBConfigTemplate, see the following sections in Container Cloud documentation:
    
    Example of a complete L2 templates configuration for cluster creation (the MetalLB configuration objects step)
    
    Examples of MetalLBConfig
    
    Examples of MetalLBConfigTemplate
  Since MOSK 23.2.2, in the Technology Preview scope, you can use BGP for announcement of external addresses of Kubernetes load-balanced services for a MOSK cluster. To configure the BGP announcement mode for MetalLB, use MetalLBConfig and MetalLBConfigTemplate objects.
  
  The use of BGP is required to announce IP addresses for load-balanced services when using MetalLB on nodes that are distributed across multiple racks. In this case, setting of rack-id labels on nodes is required, they are used in node selectors for BGPPeer, BGPAdvertisement, or both MetalLB objects to properly configure BGP connections from each node.
  Configuration example of the Machine object for the BGP announcement mode
  
  apiVersion: cluster.k8s.io/v1alpha1 kind: Machine metadata: name: test-cluster-compute-1 namespace: mosk-ns labels: cluster.sigs.k8s.io/cluster-name: test-cluster ipam/RackRef: rack-1 # reference to the "rack-1" Rack kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one spec: providerSpec: value: ... nodeLabels: - key: rack-id # node label can be used in "nodeSelectors" inside value: rack-1 # "BGPPeer" and/or "BGPAdvertisement" MetalLB objects ...
  
  Note
  
  The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.
  
  Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.
  Configuration example of the MetalLBConfigTemplate object for the BGP announcement mode
  
  apiVersion: ipam.mirantis.com/v1alpha1 kind: MetalLBConfigTemplate metadata: name: test-cluster-metallb-config-template namespace: mosk-ns labels: cluster.sigs.k8s.io/cluster-name: test-cluster kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one spec: templates: ... bgpPeers: | - name: svc-peer-1 spec: peerAddress: 10.77.42.1 peerASN: 65100 myASN: 65101 nodeSelectors: - matchLabels: rack-id: rack-1 # references the nodes having # the "rack-id=rack-1" label bgpAdvertisements: | - name: services spec: ipAddressPools: - services peers: - svc-peer-1 ...
  
  Note
  
  The kaas.mirantis.com/region label is removed from all Container Cloud and MOSK objects in 24.1.
  
  Therefore, do not add the label starting with these releases. On existing clusters updated to these releases, or if added manually, Container Cloud ignores this label.
  
  The bgpPeers and bgpAdvertisements fields are used to configure BGP announcement instead of l2Advertisements.
  The use of BGP for announcement also allows for better balancing of service traffic between cluster nodes as well as gives more configuration control and flexibility for infrastructure administrators. For configuration examples, refer to Examples of MetalLBConfigTemplate. For configuration procedure, refer to Configure BGP announcement for cluster API LB address.
- Not recommended. Configure the configInline value in the MetalLB chart of the Cluster object.
  
  Warning
  
  This option is deprecated since MOSK 23.2 and is removed during the management cluster upgrade to the Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0.
  
  Therefore, this option becomes unavailable on MOSK 23.2 clusters after the parent management cluster upgrade to 2.25.0.
- Not recommended. Configure the Subnet objects without MetalLBConfigTemplate.
  
  Warning
  
  This option is deprecated since MOSK 23.2 and is removed during the management cluster upgrade to the Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0.
  
  Therefore, this option becomes unavailable on MOSK 23.2 clusters after the parent management cluster upgrade to 2.25.0.
Caution

If the MetalLBConfig object is not used for MetalLB configuration related to address allocation and announcement for load-balanced services, then automated migration applies during cluster creation or update to MOSK 23.2.

During automated migration, the MetalLBConfig and MetalLBConfigTemplate objects are created and contents of the MetalLB chart configInline value is converted to the parameters of the MetalLBConfigTemplate object.

Any change to the configInline value made on a MOSK 23.2 cluster will be reflected in the MetalLBConfigTemplate object.

This automated migration is removed during your management cluster upgrade to the Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0, together with the possibility to use the configInline value of the MetalLB chart. After that, any changes in MetalLB configuration related to address allocation and announcement for load-balanced services are applied using the MetalLBConfig, MetalLBConfigTemplate, and Subnet objects only.
Before MOSK 23.2
Select from the following options:
- Configure Subnet objects. For details, see MetalLB configuration guidelines for subnets.
- Configure the configInline value for the MetalLB chart in the Cluster object.
- Configure both the configInline value for the MetalLB chart and Subnet objects.
  
  The resulting MetalLB address pools configuration will contain address ranges from both cluster specification and Subnet objects. All address ranges for L2 address pools will be aggregated into a single L2 address pool and sorted as strings.
Changes to be applied since MOSK 23.2

The configuration options above become deprecated since 23.2, and automated migration of MetalLB parameters applies during cluster creation or update to MOSK 23.2.

During automated migration, the MetalLBConfig and MetalLBConfigTemplate objects are created and contents of the MetalLB chart configInline value is converted to the parameters of the MetalLBConfigTemplate object.

Any change to the configInline value made on a MOSK 23.2 cluster will be reflected in the MetalLBConfigTemplate object.

This automated migration is removed during your management cluster upgrade to Container Cloud 2.25.0 together with the possibility to use the configInline value of the MetalLB chart. After that, any changes in MetalLB configuration related to address allocation and announcement for load-balanced services will be applied using the MetalLBConfigTemplate and Subnet objects only.

Verify the current MetalLB configuration:

Since MOSK 22.5

Verify the MetalLB configuration that is stored in MetalLB objects:

kubectl -n metallb-system get ipaddresspools,l2advertisements

The example system output:

NAME                                    AGE
ipaddresspool.metallb.io/default        129m
ipaddresspool.metallb.io/services-pxe   129m

NAME                                      AGE
l2advertisement.metallb.io/default        129m
l2advertisement.metallb.io/services-pxe   129m

Verify one of the listed above MetalLB objects:

kubectl -n metallb-system get <object> -o json | jq '.spec'

The example system output for ipaddresspool objects:

$ kubectl -n metallb-system get ipaddresspool.metallb.io/default -o json | jq '.spec'
{
  "addresses": [
    "10.0.11.61-10.0.11.80"
  ],
  "autoAssign": true,
  "avoidBuggyIPs": false
}
$ kubectl -n metallb-system get ipaddresspool.metallb.io/services-pxe -o json | jq '.spec'
{
  "addresses": [
    "10.0.0.61-10.0.0.70"
  ],
  "autoAssign": false,
  "avoidBuggyIPs": false
}

Before MOSK 22.5

Verify the MetalLB configuration that is stored in the ConfigMap object:

kubectl -n metallb-system get cm metallb -o jsonpath={.data.config}

An example of a successful output:

address-pools:
- name: default
  protocol: layer2
  addresses:
  - 10.0.11.61-10.0.11.80
- name: services-pxe
  protocol: layer2
  auto-assign: false
  addresses:
  - 10.0.0.61-10.0.0.70

The auto-assign parameter will be set to false for all address pools except the default one. So, a particular service will get an address from such an address pool only if the Service object has a special metallb.universe.tf/address-pool annotation that points to the specific address pool name.

Proceed to creating cluster subnets as described in Create subnets.