Configure BGP announcement for cluster API LB address¶
Available since MOSK 23.2.2TechPreview
When you create a MOSK cluster with the multi-rack topology,
where Kubernetes masters are distributed across multiple racks
without an L2 layer extension between them, you must configure
BGP announcement of the cluster API load balancer address.
For clusters where Kubernetes masters are in the same rack or with an L2 layer
extension between masters, you can configure either BGP or L2 (ARP)
announcement of the cluster API load balancer address.
The L2 (ARP) announcement is used by default and its configuration is covered
in Create a managed bare metal cluster.
Caution
Create Rack and MultiRackCluster objects, which are
described in the below procedure, before initiating the provisioning
of master nodes to ensure that both BGP and netplan configurations
are applied simultaneously during the provisioning process.
To enable the use of BGP announcement for the cluster API LB address:
In the Cluster object, set the useBGPAnnouncement parameter
to true:
spec:providerSpec:value:useBGPAnnouncement:true
Create the MultiRackCluster object that is mandatory when configuring
BGP announcement for the cluster API LB address. This object enables you
to set cluster-wide parameters for configuration of BGP announcement.
In this scenario, the MultiRackCluster object must be bound to the
corresponding Cluster object using the
cluster.sigs.k8s.io/cluster-name label.
Container Cloud uses the bird BGP daemon for announcement of the cluster
API LB address. For this reason, set the corresponding
bgpdConfigFileName and bgpdConfigFilePath parameters in the
MultiRackCluster object, so that bird can locate the configuration
file. For details, see the configuration example below.
The bgpdConfigTemplate object contains the default configuration file
template for the bird BGP daemon, which you can override in Rack
objects.
The defaultPeer parameter contains default parameters of the BGP
connection from master nodes to infrastructure BGP peers, which you can
override in Rack objects.
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.
Create the Rack object(s). This object is mandatory when configuring
BGP announcement for the cluster API LB address and it allows you
to configure BGP announcement parameters for each rack.
In this scenario, Rack objects must be bound to Machine objects
corresponding to master nodes of the cluster.
Each Rack object describes the configuration for the bird BGP
daemon used to announce the cluster API LB address from a particular
master node or from several master nodes in the same rack.
Set a reference to the Rack object used to configure the bird BGP
daemon for a particular master node to announce the cluster API LB IP:
Since MOSK 25.1
In the Machine objects for all master nodes, set the ipam/RackRef
label with the value equal to the name of the corresponding Rack
object. For example:
apiVersion:cluster.k8s.io/v1alpha1kind:Machinemetadata:labels:ipam/RackRef:rack-master-1# reference to the "rack-master-1" Rack...
Before MOSK 25.1. (deprecated)
In the BareMetalHost objects for all cluster nodes, set the
ipam.mirantis.com/rack-ref annotation with the value equal to the name
of the corresponding Rack object. For example:
apiVersion:metal3.io/v1alpha1kind:BareMetalHostmetadata:annotations:ipam.mirantis.com/rack-ref:rack-master-1# reference to the "rack-master-1" Rack...
Optional. Using the Machine object, define the rack-id node label
that is not used for BGP announcement of the cluster API LB IP but
can be used for MetalLB.
The rack-id node label is required for MetalLB node selectors when
MetalLB is used to announce LB IP addresses on nodes that are distributed
across multiple racks. In this scenario, the L2 (ARP) announcement mode
cannot be used for MetalLB because master nodes are in different L2
segments. So, the BGP announcement mode must be used for MetalLB, and node
selectors are required to properly configure BGP connections from each node.
See Configure MetalLB for details.
The L2Template object includes the lo interface configuration
to set the IP address for the bird BGP daemon that will be advertised
as the cluster API LB address. The {{ cluster_api_lb_ip }}
function is used in npTemplate to obtain the cluster API LB address
value.
Configuration example for Rack
apiVersion:ipam.mirantis.com/v1alpha1kind:Rackmetadata:name:rack-master-1namespace:mosk-nslabels:cluster.sigs.k8s.io/cluster-name:test-clusterkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onespec:bgpdConfigTemplate:|# optional...peeringMap:lcm-rack-control-1:peers:-neighborIP:10.77.31.2# "localASN" & "neighborASN" are taken from-neighborIP:10.77.31.3# "MultiRackCluster.spec.defaultPeer" if# not set here
Note
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.
Configuration example for Machine
apiVersion:cluster.k8s.io/v1alpha1kind:Machinemetadata:name:test-cluster-master-1namespace:mosk-nsannotations:metal3.io/BareMetalHost:mosk-ns/test-cluster-master-1labels:cluster.sigs.k8s.io/cluster-name:test-clustercluster.sigs.k8s.io/control-plane:controlplanehostlabel.bm.kaas.mirantis.com/controlplane:controlplaneipam/RackRef:rack-master-1# reference to the "rack-master-1" Rackkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onespec:providerSpec:value:kind:BareMetalMachineProviderSpecapiVersion:baremetal.k8s.io/v1alpha1hostSelector:matchLabels:kaas.mirantis.com/baremetalhost-id:test-cluster-master-1l2TemplateSelector:name:test-cluster-master-1nodeLabels:# optional. it is not used for BGP announcement-key:rack-id# of the cluster API LB IP but it can be usedvalue:rack-master-1# for MetalLB if "nodeSelectors" are required...
Note
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.
Configuration example for L2Template
apiVersion:ipam.mirantis.com/v1alpha1kind:L2Templatemetadata:labels:cluster.sigs.k8s.io/cluster-name:test-clusterkaas.mirantis.com/provider:baremetalkaas.mirantis.com/region:region-onename:test-cluster-master-1namespace:mosk-nsspec:...l3Layout:-subnetName:lcm-rack-control-1# this network is referencedscope:namespace# in the "rack-master-1" Rack-subnetName:ext-rack-control-1# optional. this network is usedscope:namespace# for k8s services traffic and# MetalLB BGP connections...npTemplate:|...ethernets:lo:addresses:- {{ cluster_api_lb_ip }} # function for cluster API LB IPdhcp4: falsedhcp6: false...
Note
The kaas.mirantis.com/region label is removed from all
Container Cloud and MOSK objects in 24.1.
Therefore, do not add the label starting with these releases. On existing
clusters updated to these releases, or if added manually, Container Cloud
ignores this label.