This section describes how to set up and verify MetalLB parameters during
configuration of subnets for a managed cluster creation.
Caution
This section also applies to the bootstrap procedure of a
management or regional cluster with the following differences:
Instead of the Cluster object, configure
templates/bm/cluster.yaml.template.
Instead of the MetalLBConfig object, configure
templates/bm/metallbconfig.yaml.template.
Instead of creating specific IPAM objects such as Subnet and
MetalLBConfigTemplate, add their configuration to
templates/bm/ipam-objects.yaml.template.
The Kubernetes objects described below are created for a management or
regional cluster from template files during bootstrap.
To configure MetalLB:
Optional. Configure parameters related to MetalLB components life cycle
such as deployment and update using the metallb Helm chart values in
the Clusterspec section. For example:
Configure the MetalLB parameters related to LB address allocation and
announcement for cluster services:
Since Container Cloud 2.24.0
Select from the following options:
Recommended. Default. Create MetalLBConfig and
MetalLBConfigTemplate objects. This method allows using the
Subnet object to define MetalLB address pools. For details, see API
documentation: MetalLBConfig and MetalLBConfigTemplate.
For configuration examples that use MetalLBConfig and
MetalLBConfigTemplate, see the following sections:
For managed clusters, this configuration method is
available as Technology Preview and will become generally available in
one of the following Container Cloud releases.
Since Container Cloud 2.24.4, in the Technology Preview scope, you can
use BGP for announcement of external addresses of Kubernetes
load-balanced services for managed clusters. To configure the BGP
announcement mode for MetalLB, use MetalLBConfig and
MetalLBConfigTemplate objects.
The use of BGP is required to announce services LB IP addresses
when using MetalLB on nodes that are distributed across multiple racks.
In this case, setting of rack-id labels on nodes is required,
they are used in node selectors for BGPPeer, BGPAdvertisement,
or both MetalLB objects to properly configure BGP connections from each
node.
Configuration example of the Machine object for the
BGP announcement mode
apiVersion:cluster.k8s.io/v1alpha1kind:Machinemetadata:name:test-cluster-compute-1namespace:managed-nslabels:cluster.sigs.k8s.io/cluster-name:test-clusteripam/RackRef:rack-1# reference to the "rack-1" Rackkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onespec:providerSpec:value:...nodeLabels:-key:rack-id# node label can be used in "nodeSelectors" insidevalue:rack-1# "BGPPeer" and/or "BGPAdvertisement" MetalLB objects...
Configuration example of the MetalLBConfigTemplate
object for the BGP announcement mode
apiVersion:ipam.mirantis.com/v1alpha1kind:MetalLBConfigTemplatemetadata:name:test-cluster-metallb-config-templatenamespace:managed-nslabels:cluster.sigs.k8s.io/cluster-name:test-clusterkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onespec:templates:...bgpPeers:|- name: svc-peer-1spec:peerAddress: 10.77.42.1peerASN: 65100myASN: 65101nodeSelectors:- matchLabels:rack-id: rack-1 # references the nodes having# the "rack-id=rack-1" labelbgpAdvertisements:|- name: servicesspec:ipAddressPools:- servicespeers:- svc-peer-1...
The bgpPeers and bgpAdvertisements fields are used to
configure BGP announcement instead of l2Advertisements.
The use of BGP for announcement also allows for better balancing
of service traffic between cluster nodes as well as gives more
configuration control and flexibility for infrastructure administrators.
For configuration examples, refer to MetalLB configuration examples.
For configuration procedure, refer to Configure BGP announcement for cluster API LB address.
The following rules and requirements apply to configuration of
MetalLBConfig and MetalLBConfigTemplate:
Define one MetalLBConfig and MetalLBConfigTemplate object per
cluster.
Define the following mandatory labels:
cluster.sigs.k8s.io/cluster-name
Specifies the cluster name where the MetalLB address pool is used.
kaas.mirantis.com/region
Specifies the region name of the cluster where the MetalLB address pool is
used.
kaas.mirantis.com/provider
Specifies the provider of the cluster where the MetalLB address pool is used.
You can use MetalLBConfig without MetalLBConfigTemplate
but not the opposite way.
You can use the MetalLBConfig and MetalLBConfigTemplate objects
to optimize the announcement of the services LB addresses using
the interfaces selector for the l2Advertisements object.
This selector allows for announcement of addresses only on selected
host interfaces. For details, see API Reference:
MetalLBConfigTemplate spec.
Mirantis recommends this configuration if nodes use separate host
networks for different types of traffic. The pros of such
configuration are as follows: less spam on other interfaces and
networks, limited chances to reach services LB addresses from
irrelevant interfaces and networks.
When using MetalLBConfigTemplate:
MetalLBConfig must reference MetalLBConfigTemplate by name:
spec:templateName:<managed-metallb-template>
You can use Subnet objects for defining MetalLB address pools.
Refer to MetalLB configuration guidelines for subnets for guidelines on configuring
MetalLB address pools using Subnet objects.
The use of MetalLBConfigTemplate object gives more flexibility
when describing MetalLB configuration as it allows to use variables,
functions, and the Go template expressions inside object templates.
Intersection of IP address ranges within any single MetalLB address
pool is not allowed.
At least one MetalLB address pool must have the auto-assign policy
enabled so that unannotated services can have load balancer IPs
allocated for them.
When configuring multiple address pools with the auto-assign policy
enabled, keep in mind that it is not determined in advance which pool
of those multiple address pools is used to allocate an IP for a
particular unannotated service.
Deprecated since Container Cloud 2.24.0. Configure the configInline
value in the MetalLB chart of the Cluster object. This functionality
will be removed in one of the following releases.
Deprecated since Container Cloud 2.24.0. Configure the Subnet
objects without MetalLBConfigTemplate. This functionality will be
removed in one of the following releases.
Caution
If the MetalLBConfig object is not used for MetalLB
configuration related to address allocation and announcement of services
LB, then automated migration applies during creation of clusters of any
type or cluster update to Container Cloud 2.24.0.
During automated migration, the MetalLBConfig and
MetalLBConfigTemplate objects are created and contents of the MetalLB
chart configInline value is converted to the parameters of the
MetalLBConfigTemplate object.
Any change to the configInline value made on a Container Cloud 2.24.0
cluster will be reflected in the MetalLBConfigTemplate object.
This automated migration will be removed in one of the following releases
together with the possibility to use the configInline value of the
MetalLB chart. After that, any changes in MetalLB configuration related
to address allocation and announcement of services LB will be applied
using the MetalLBConfigTemplate and Subnet objects only.
Before Container Cloud 2.24.0
Select from the following options:
Configure the configInline value for the MetalLB chart in the
Cluster object.
Caution
The use of configInline value for the MetalLB chart in
the Cluster object is deprecated since Container Cloud 2.24.0 and
will be removed in one of the following releases.
Configure both the configInline value for the MetalLB chart and
Subnet objects.
The resulting MetalLB address pools configuration will contain address
ranges from both cluster specification and Subnet objects.
All address ranges for L2 address pools will be aggregated into a single
L2 address pool and sorted as strings.
Caution
The use of configInline value for the MetalLB chart
in the Cluster object is deprecated since Container Cloud 2.24.0
and will be removed in one of the following releases.
Technology Preview since Container Cloud 2.21.1 that applies to managed
clusters only. Configure MetalLB address pools and advertisement
parameters using the MetalLBConfig object. If you plan to use the
MetalLBConfig object in future, refer to the configuration steps
for Container Cloud 2.24.0 or later.
You can use this object instead of Subnet objects and MetalLB chart
values in the Cluster object. Do not mix both configuration methods.
Verify that no subnets with the ipam/SVC-MetalLB label are
defined for a particular managed cluster.
Verify that the configInline value is absent for the
MetalLB chart in the Cluster object. Other fields in values
for the MetalLB chart are allowed.
Create the MetalLBConfig object that must reference the target
cluster name. Define one MetalLBConfig object per cluster.
Add MetalLB configuration parameters to the MetalLBConfig object
definition as described in MetalLBConfig.
Verify the current MetalLB configuration:
Since Container Cloud 2.21.0
Verify the MetalLB configuration that is stored in MetalLB objects:
The auto-assign parameter will be set to false for all address
pools except the default one. So, a particular service will get an
address from such an address pool only if the Service object has a
special metallb.universe.tf/address-pool annotation that points to
the specific address pool name.
Note
It is expected that every Container Cloud service on a management
and regional cluster will be assigned to one of the address pools.
Current consideration is to have two MetalLB address pools:
services-pxe is a reserved address pool name to use for
the Container Cloud services in the PXE network (Ironic API,
HTTP server, caching server).
default is an address pool to use for all other Container
Cloud services in the management network. No annotation
is required on the Service objects in this case.