This section describes how to set up and verify MetalLB parameters during
configuration of subnets for a MOSK cluster creation.
Configuration rules for ‘MetalLBConfig’ and ‘MetalLBConfigTemplate’ objects¶
Caution
The use of the MetalLBConfig object is mandatory after your
management cluster upgrade to the Cluster release 17.0.0.
The following rules and requirements apply to configuration of the
MetalLBConfig and MetalLBConfigTemplate objects:
Define one MetalLBConfig and MetalLBConfigTemplate object per
cluster.
Define the following mandatory labels:
cluster.sigs.k8s.io/cluster-name
Specifies the cluster name where the MetalLB address pool is used.
kaas.mirantis.com/region
Specifies the region name of the cluster where the MetalLB address pool is
used.
kaas.mirantis.com/provider
Specifies the provider of the cluster where the MetalLB address pool is used.
Note
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.
You can use MetalLBConfig without MetalLBConfigTemplate
but not the opposite way.
You can use the MetalLBConfig and MetalLBConfigTemplate objects
to optimize address announcement for load-balanced services using the
interfaces selector for the l2Advertisements object.
This selector allows for announcing addresses only on selected
host interfaces. For details, see Container Cloud API Reference:
MetalLBConfigTemplate spec.
Mirantis recommends this configuration if nodes use separate host
networks for different types of traffic. The advantages of such
configuration include reduced congestion on other interfaces and
networks, as well as limited exposure of addresses of load-balanced
services to irrelevant interfaces and networks.
When using MetalLBConfigTemplate:
MetalLBConfig must reference MetalLBConfigTemplate by name:
spec:templateName:<managed-metallb-template>
You can use Subnet objects for defining MetalLB address pools.
Refer to MetalLB configuration guidelines for subnets for guidelines on configuring
MetalLB address pools using Subnet objects.
The use of MetalLBConfigTemplate object gives more flexibility
when describing MetalLB configuration as it allows using variables,
functions, and the Go template expressions inside object templates.
Intersection of IP address ranges within any single MetalLB address pool
is not allowed.
At least one MetalLB address pool must have the auto-assign policy enabled
so that unannotated services can have load balancer IPs allocated for them.
When configuring multiple address pools with the auto-assign policy enabled,
keep in mind that it is not determined in advance which pool of those
multiple address pools is used to allocate an IP for a particular
unannotated service.
Optional. Configure parameters related to MetalLB components life cycle
such as deployment and update using the metallb Helm chart values in
the Clusterspec section. For example:
Configure the MetalLB parameters related to IP address allocation and
announcement for load-balanced cluster services:
Since MOSK 23.2
Select from the following options:
Mandatory after a management cluster upgrade to the Cluster release
17.0.0. Recommended and default since MOSK 23.2 in
the Technology Preview scope.
Create MetalLBConfig and MetalLBConfigTemplate objects.
This method allows using the Subnet object to define MetalLB
address pools.
Since MOSK 23.2.2, in the Technology Preview scope,
you can use BGP for announcement of external addresses of Kubernetes
load-balanced services for a MOSK cluster.
To configure the BGP announcement mode for MetalLB, use
MetalLBConfig and MetalLBConfigTemplate objects.
The use of BGP is required to announce IP addresses for load-balanced
services when using MetalLB on nodes that are distributed across
multiple racks. In this case, setting of rack-id labels on nodes
is required, they are used in node selectors for BGPPeer,
BGPAdvertisement, or both MetalLB objects to properly configure
BGP connections from each node.
Configuration example of the Machine object for the
BGP announcement mode
apiVersion:cluster.k8s.io/v1alpha1kind:Machinemetadata:name:test-cluster-compute-1namespace:mosk-nslabels:cluster.sigs.k8s.io/cluster-name:test-clusteripam/RackRef:rack-1# reference to the "rack-1" Rackkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onespec:providerSpec:value:...nodeLabels:-key:rack-id# node label can be used in "nodeSelectors" insidevalue:rack-1# "BGPPeer" and/or "BGPAdvertisement" MetalLB objects...
Note
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.
Configuration example of the MetalLBConfigTemplate
object for the BGP announcement mode
apiVersion:ipam.mirantis.com/v1alpha1kind:MetalLBConfigTemplatemetadata:name:test-cluster-metallb-config-templatenamespace:mosk-nslabels:cluster.sigs.k8s.io/cluster-name:test-clusterkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onespec:templates:...bgpPeers:|- name: svc-peer-1spec:peerAddress: 10.77.42.1peerASN: 65100myASN: 65101nodeSelectors:- matchLabels:rack-id: rack-1 # references the nodes having# the "rack-id=rack-1" labelbgpAdvertisements:|- name: servicesspec:ipAddressPools:- servicespeers:- svc-peer-1...
Note
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.
The bgpPeers and bgpAdvertisements fields are used to
configure BGP announcement instead of l2Advertisements.
The use of BGP for announcement also allows for better balancing
of service traffic between cluster nodes as well as gives more
configuration control and flexibility for infrastructure administrators.
For configuration examples, refer to Examples of MetalLBConfigTemplate.
For configuration procedure, refer to Configure BGP announcement for cluster API LB address.
Not recommended. Configure the configInline value in the MetalLB
chart of the Cluster object.
Warning
This option is deprecated since MOSK
23.2 and is removed during the management cluster upgrade to the
Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0.
Therefore, this option becomes unavailable on MOSK
23.2 clusters after the parent management cluster upgrade to 2.25.0.
Not recommended. Configure the Subnet objects without
MetalLBConfigTemplate.
Warning
This option is deprecated since MOSK
23.2 and is removed during the management cluster upgrade to the
Cluster release 17.0.0, which is introduced in Container Cloud 2.25.0.
Therefore, this option becomes unavailable on MOSK
23.2 clusters after the parent management cluster upgrade to 2.25.0.
Caution
If the MetalLBConfig object is not used for MetalLB
configuration related to address allocation and announcement for
load-balanced services, then automated migration applies during
cluster creation or update to MOSK 23.2.
During automated migration, the MetalLBConfig and
MetalLBConfigTemplate objects are created and contents of the MetalLB
chart configInline value is converted to the parameters of the
MetalLBConfigTemplate object.
Any change to the configInline value made on a
MOSK 23.2 cluster will be reflected in the
MetalLBConfigTemplate object.
This automated migration is removed during your management cluster
upgrade to the Cluster release 17.0.0, which is introduced in Container
Cloud 2.25.0, together with the possibility to use the configInline
value of the MetalLB chart. After that, any changes in MetalLB
configuration related to address allocation and announcement for
load-balanced services are applied using the MetalLBConfig,
MetalLBConfigTemplate, and Subnet objects only.
Configure the configInline value for the MetalLB chart in the
Cluster object.
Configure both the configInline value for the MetalLB chart and
Subnet objects.
The resulting MetalLB address pools configuration will contain address
ranges from both cluster specification and Subnet objects.
All address ranges for L2 address pools will be aggregated into a single
L2 address pool and sorted as strings.
Changes to be applied since MOSK 23.2
The configuration options above become deprecated since 23.2, and
automated migration of MetalLB parameters applies during cluster creation
or update to MOSK 23.2.
During automated migration, the MetalLBConfig and
MetalLBConfigTemplate objects are created and contents of the
MetalLB chart configInline value is converted to the parameters of
the MetalLBConfigTemplate object.
Any change to the configInline value made on a
MOSK 23.2 cluster will be reflected in the
MetalLBConfigTemplate object.
This automated migration is removed during your management cluster
upgrade to Container Cloud 2.25.0 together with the possibility to use
the configInline value of the MetalLB chart. After that, any
changes in MetalLB configuration related to address allocation and
announcement for load-balanced services will be applied using the
MetalLBConfigTemplate and Subnet objects only.
Verify the current MetalLB configuration:
Since MOSK 22.5
Verify the MetalLB configuration that is stored in MetalLB objects:
The auto-assign parameter will be set to false for all address
pools except the default one. So, a particular service will get an
address from such an address pool only if the Service object has a
special metallb.universe.tf/address-pool annotation that points to
the specific address pool name.