MultiRackCluster¶
TechPreview Available since 2.24.4
This section describes the MultiRackCluster
resource used in the
Container Cloud API.
When you create a bare metal managed cluster with a multi-rack topology,
where Kubernetes masters are distributed across multiple racks
without L2 layer extension between them, the MultiRackCluster
resource
allows you to set cluster-wide parameters for configuration of the
BGP announcement of the cluster API load balancer address.
In this scenario, the MultiRackCluster
object must be bound to the
Cluster
object.
The MultiRackCluster
object is generally used for a particular cluster
in conjunction with Rack
objects described in Rack.
For demonstration purposes, the Container Cloud MultiRackCluster
custom resource (CR) description is split into the following major sections:
MultiRackCluster metadata¶
The Container Cloud MultiRackCluster
CR metadata contains the following
fields:
apiVersion
API version of the object that is
ipam.mirantis.com/v1alpha1
.
kind
Object type that is
MultiRackCluster
.
metadata
The
metadata
field contains the following subfields:name
Name of the
MultiRackCluster
object.
namespace
Container Cloud project (Kubernetes namespace) in which the object was created.
labels
Key-value pairs that are attached to the object:
cluster.sigs.k8s.io/cluster-name
Cluster
object name that thisMultiRackCluster
object is applied to. To enable the use of BGP announcement for the cluster API LB address, set theuseBGPAnnouncement
parameter in theCluster
object totrue
:spec: providerSpec: value: useBGPAnnouncement: true
kaas.mirantis.com/provider
Provider name that is
baremetal
.
kaas.mirantis.com/region
Region name.
Note
The
kaas.mirantis.com/region
label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.
Warning
Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.
The MultiRackCluster
metadata configuration example:
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
name: multirack-test-cluster
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
MultiRackCluster spec¶
The spec
field of the MultiRackCluster
resource describes the desired
state of the object. It contains the following fields:
bgpdConfigFileName
Name of the configuration file for the BGP daemon (bird). Recommended value is
bird.conf
.
bgpdConfigFilePath
Path to the directory where the configuration file for the BGP daemon (bird) is added. The recommended value is
/etc/bird
.
bgpdConfigTemplate
Optional. Configuration text file template for the BGP daemon (bird) configuration file where you can use
go
template constructs and the following variables:RouterID
,LocalIP
Local IP on the given network, which is a key in the
Rack.spec.peeringMap
dictionary, for a given node. You can use it, for example, in therouter id {{$.RouterID}};
instruction.
LocalASN
Local AS number.
NeighborASN
Neighbor AS number.
NeighborIP
Neighbor IP address. Its values are taken from
Rack.spec.peeringMap
, it can be used only inside therange
iteration through theNeighbors
list.
Neighbors
List of peers in the given network and node. It can be iterated through the
range
statement in thego
template.
Values for
LocalASN
andNeighborASN
are taken from:MultiRackCluster.defaultPeer
- if not used as a field inside therange
iteration through theNeighbors
list.Corresponding values of
Rack.spec.peeringMap
- if used as a field inside therange
iteration through theNeighbors
list.
This template can be overridden using the
Rack
objects. For details, see Rack spec.
defaultPeer
Configuration parameters for the default BGP peer. These parameters will be used in rendering of the configuration file for BGP daemon from the template if they are not overridden for a particular rack or network using
Rack
objects. For details, see Rack spec.localASN
Mandatory. Local AS number.
neighborASN
Mandatory. Neighbor AS number.
neighborIP
Reserved. Neighbor IP address. Leave it as an empty string.
password
Optional. Neighbor password. If not set, you can hardcode it in
bgpdConfigTemplate
. It is required for MD5 authentication between BGP peers.
Configuration examples:
Since Cluster releases 17.1.0 and 16.1.0 for bird
v2.x
spec:
bgpdConfigFileName: bird.conf
bgpdConfigFilePath: /etc/bird
bgpdConfigTemplate: |
protocol device {
}
#
protocol direct {
interface "lo";
ipv4;
}
#
protocol kernel {
ipv4 {
export all;
};
}
#
{{range $i, $peer := .Neighbors}}
protocol bgp 'bgp_peer_{{$i}}' {
local port 1179 as {{.LocalASN}};
neighbor {{.NeighborIP}} as {{.NeighborASN}};
ipv4 {
import none;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
};
}
{{end}}
defaultPeer:
localASN: 65101
neighborASN: 65100
neighborIP: ""
Before Cluster releases 17.1.0 and 16.1.0 for bird
v1.x
spec:
bgpdConfigFileName: bird.conf
bgpdConfigFilePath: /etc/bird
bgpdConfigTemplate: |
listen bgp port 1179;
protocol device {
}
#
protocol direct {
interface "lo";
}
#
protocol kernel {
export all;
}
#
{{range $i, $peer := .Neighbors}}
protocol bgp 'bgp_peer_{{$i}}' {
local as {{.LocalASN}};
neighbor {{.NeighborIP}} as {{.NeighborASN}};
import all;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
}
{{end}}
defaultPeer:
localASN: 65101
neighborASN: 65100
neighborIP: ""
MultiRackCluster status¶
The status
field of the MultiRackCluster
resource reflects the actual
state of the MultiRackCluster
object and contains the following fields:
state
Since 2.23.0Message that reflects the current status of the resource. The list of possible values includes the following:
OK
- object is operational.ERR
- object is non-operational. This status has a detailed description in themessages
list.TERM
- object was deleted and is terminating.
messages
Since 2.23.0List of error or warning messages if the object state is
ERR
.
objCreated
Date, time, and IPAM version of the resource creation.
objStatusUpdated
Date, time, and IPAM version of the last update of the
status
field in the resource.
objUpdated
Date, time, and IPAM version of the last resource update.
Configuration example:
status:
checksums:
annotations: sha256:38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed
labels: sha256:d8f8eacf487d57c22ca0ace29bd156c66941a373b5e707d671dc151959a64ce7
spec: sha256:66b5d28215bdd36723fe6230359977fbede828906c6ae96b5129a972f1fa51e9
objCreated: 2023-08-11T12:25:21.00000Z by v6.5.999-20230810-155553-2497818
objStatusUpdated: 2023-08-11T12:32:58.11966Z by v6.5.999-20230810-155553-2497818
objUpdated: 2023-08-11T12:32:57.32036Z by v6.5.999-20230810-155553-2497818
state: OK
MultiRackCluster and Rack usage examples¶
The following configuration examples of several bare metal objects illustrate how to configure BGP announcement of the load balancer address used to expose the cluster API.
Single rack configuration example¶
In the following example, all master nodes are in a single rack. One Rack
object is required in this case for master nodes. Some worker nodes can
coexist in the same rack with master nodes or occupy separate racks. It is
implied that the useBGPAnnouncement
parameter is set to true
in the
corresponding Cluster
object.
Configuration example for MultiRackCluster
Since Cluster releases 17.1.0 and 16.1.0 for bird
v2.x:
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
name: multirack-test-cluster
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
spec:
bgpdConfigFileName: bird.conf
bgpdConfigFilePath: /etc/bird
bgpdConfigTemplate: |
protocol device {
}
#
protocol direct {
interface "lo";
ipv4;
}
#
protocol kernel {
ipv4 {
export all;
};
}
#
{{range $i, $peer := .Neighbors}}
protocol bgp 'bgp_peer_{{$i}}' {
local port 1179 as {{.LocalASN}};
neighbor {{.NeighborIP}} as {{.NeighborASN}};
ipv4 {
import none;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
};
}
{{end}}
defaultPeer:
localASN: 65101
neighborASN: 65100
neighborIP: ""
Before Cluster releases 17.1.0 and 16.1.0 for bird
v1.x:
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
name: multirack-test-cluster
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
spec:
bgpdConfigFileName: bird.conf
bgpdConfigFilePath: /etc/bird
bgpdConfigTemplate: |
listen bgp port 1179;
protocol device {
}
#
protocol direct {
interface "lo";
}
#
protocol kernel {
export all;
}
#
{{range $i, $peer := .Neighbors}}
protocol bgp 'bgp_peer_{{$i}}' {
local as {{.LocalASN}};
neighbor {{.NeighborIP}} as {{.NeighborASN}};
import all;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
}
{{end}}
defaultPeer:
localASN: 65101
neighborASN: 65100
neighborIP: ""
Configuration example for Rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
name: rack-master
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
spec:
peeringMap:
lcm-rack-control:
peers:
- neighborIP: 10.77.31.1 # "localASN" and "neighborASN" are taken from
- neighborIP: 10.77.37.1 # "MultiRackCluster.spec.defaultPeer"
# if not set here
Configuration example for Machine
# "Machine" templates for "test-cluster-master-2" and "test-cluster-master-3"
# differ only in BMH selectors in this example.
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
name: test-cluster-master-1
namespace: managed-ns
annotations:
metal3.io/BareMetalHost: managed-ns/test-cluster-master-1
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
cluster.sigs.k8s.io/control-plane: controlplane
hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
ipam/RackRef: rack-master # used to connect "IpamHost" to "Rack" objects, so that
# BGP parameters can be obtained from "Rack" to
# render BGP configuration for the given "IpamHost" object
kaas.mirantis.com/provider: baremetal
spec:
providerSpec:
value:
kind: BareMetalMachineProviderSpec
apiVersion: baremetal.k8s.io/v1alpha1
hostSelector:
matchLabels:
kaas.mirantis.com/baremetalhost-id: test-cluster-master-1
l2TemplateSelector:
name: test-cluster-master
Configuration example for L2Template
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: test-cluster-master
namespace: managed-ns
spec:
...
l3Layout:
- subnetName: lcm-rack-control # this network is referenced in "rack-master" Rack
scope: namespace
...
npTemplate: |
...
ethernets:
lo:
addresses:
- {{ cluster_api_lb_ip }} # function for cluster API LB IP
dhcp4: false
dhcp6: false
...
After the objects are created and nodes are provisioned, the IpamHost
objects will have BGP daemon configuration files in their status
fields.
For example:
Configuration example for IpamHost
apiVersion: ipam.mirantis.com/v1alpha1
kind: IpamHost
...
status:
...
netconfigFiles:
- content: bGlzdGVuIGJncCBwb3J0IDExNzk7CnByb3RvY29sIGRldmljZSB7Cn0KIwpwcm90b2NvbCBkaXJlY3QgewogIGludGVyZmFjZSAibG8iOwp9CiMKcHJvdG9jb2wga2VybmVsIHsKICBleHBvcnQgYWxsOwp9CiMKCnByb3RvY29sIGJncCAnYmdwX3BlZXJfMCcgewogIGxvY2FsIGFzIDY1MTAxOwogIG5laWdoYm9yIDEwLjc3LjMxLjEgYXMgNjUxMDA7CiAgaW1wb3J0IGFsbDsKICBleHBvcnQgZmlsdGVyIHsKICAgIGlmIGRlc3QgPSBSVERfVU5SRUFDSEFCTEUgdGhlbiB7CiAgICAgIHJlamVjdDsKICAgIH0KICAgIGFjY2VwdDsKICB9Owp9Cgpwcm90b2NvbCBiZ3AgJ2JncF9wZWVyXzEnIHsKICBsb2NhbCBhcyA2NTEwMTsKICBuZWlnaGJvciAxMC43Ny4zNy4xIGFzIDY1MTAwOwogIGltcG9ydCBhbGw7CiAgZXhwb3J0IGZpbHRlciB7CiAgICBpZiBkZXN0ID0gUlREX1VOUkVBQ0hBQkxFIHRoZW4gewogICAgICByZWplY3Q7CiAgICB9CiAgICBhY2NlcHQ7CiAgfTsKfQoK
path: /etc/bird/bird.conf
- content: ...
path: /etc/netplan/60-kaas-lcm-netplan.yaml
netconfigFilesStates:
/etc/bird/bird.conf: 'OK: 2023-08-17T08:00:58.96140Z 25cde040e898fd5bf5b28aacb12f046b4adb510570ecf7d7fa5a8467fa4724ec'
/etc/netplan/60-kaas-lcm-netplan.yaml: 'OK: 2023-08-11T12:33:24.54439Z 37ac6e9fe13e5969f35c20c615d96b4ed156341c25e410e95831794128601e01'
...
You can decode /etc/bird/bird.conf
contents and verify the configuration:
echo "<<base64-string>>" | base64 -d
The following system output applies to the above configuration examples:
Configuration example for the decoded bird.conf
Since Cluster releases 17.1.0 and 16.1.0 for bird
v2.x:
protocol device {
}
#
protocol direct {
interface "lo";
ipv4;
}
#
protocol kernel {
ipv4 {
export all;
};
}
#
protocol bgp 'bgp_peer_0' {
local port 1179 as 65101;
neighbor 10.77.31.1 as 65100;
ipv4 {
import none;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
};
}
protocol bgp 'bgp_peer_1' {
local port 1179 as 65101;
neighbor 10.77.37.1 as 65100;
ipv4 {
import none;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
};
}
Before Cluster releases 17.1.0 and 16.1.0 for bird
v1.x:
listen bgp port 1179;
protocol device {
}
#
protocol direct {
interface "lo";
}
#
protocol kernel {
export all;
}
#
protocol bgp 'bgp_peer_0' {
local as 65101;
neighbor 10.77.31.1 as 65100;
import all;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
}
protocol bgp 'bgp_peer_1' {
local as 65101;
neighbor 10.77.37.1 as 65100;
import all;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
}
BGP daemon configuration files are copied from IpamHost.status
to the corresponding LCMMachine
object the same way as it is done for
netplan configuration files. Then, the configuration files are written to the
corresponding node by the LCM-Agent.
Multiple rack configuration example¶
In the following configuration example, each master node is located in its own
rack. Three Rack
objects are required in this case for master nodes.
Some worker nodes can coexist in the same racks with master nodes or occupy
separate racks.
Only objects that are required to show configuration for BGP announcement
of the cluster API load balancer address are provided here.
For the description of Rack
, MetalLBConfig
,
and other objects that are required for MetalLB configuration in this scenario,
refer to Configuration example for using BGP announcement.
It is implied that the useBGPAnnouncement
parameter is set to true
in the corresponding Cluster
object.
Configuration example for MultiRackCluster
Since Cluster releases 17.1.0 and 16.1.0 for bird
v2.x:
# It is the same object as in the single rack example.
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
name: multirack-test-cluster
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
spec:
bgpdConfigFileName: bird.conf
bgpdConfigFilePath: /etc/bird
bgpdConfigTemplate: |
protocol device {
}
#
protocol direct {
interface "lo";
ipv4;
}
#
protocol kernel {
ipv4 {
export all;
};
}
#
{{range $i, $peer := .Neighbors}}
protocol bgp 'bgp_peer_{{$i}}' {
local port 1179 as {{.LocalASN}};
neighbor {{.NeighborIP}} as {{.NeighborASN}};
ipv4 {
import none;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
};
}
{{end}}
defaultPeer:
localASN: 65101
neighborASN: 65100
neighborIP: ""
Before Cluster releases 17.1.0 and 16.1.0 for bird
v1.x:
# It is the same object as in the single rack example.
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
name: multirack-test-cluster
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
spec:
bgpdConfigFileName: bird.conf
bgpdConfigFilePath: /etc/bird
bgpdConfigTemplate: |
listen bgp port 1179;
protocol device {
}
#
protocol direct {
interface "lo";
}
#
protocol kernel {
export all;
}
#
{{range $i, $peer := .Neighbors}}
protocol bgp 'bgp_peer_{{$i}}' {
local as {{.LocalASN}};
neighbor {{.NeighborIP}} as {{.NeighborASN}};
import all;
export filter {
if dest = RTD_UNREACHABLE then {
reject;
}
accept;
};
}
{{end}}
defaultPeer:
localASN: 65101
neighborASN: 65100
neighborIP: ""
The following Rack
objects differ in neighbor IP addresses and in the
network (L3 subnet) used for BGP connection to announce the cluster API LB IP
and for cluster API traffic.
Configuration example for Rack
1
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
name: rack-master-1
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
spec:
peeringMap:
lcm-rack-control-1:
peers:
- neighborIP: 10.77.31.2 # "localASN" and "neighborASN" are taken from
- neighborIP: 10.77.31.3 # "MultiRackCluster.spec.defaultPeer" if
# not set here
Configuration example for Rack
2
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
name: rack-master-2
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
spec:
peeringMap:
lcm-rack-control-2:
peers:
- neighborIP: 10.77.32.2 # "localASN" and "neighborASN" are taken from
- neighborIP: 10.77.32.3 # "MultiRackCluster.spec.defaultPeer" if
# not set here
Configuration example for Rack
3
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
name: rack-master-3
namespace: managed-ns
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
spec:
peeringMap:
lcm-rack-control-3:
peers:
- neighborIP: 10.77.33.2 # "localASN" and "neighborASN" are taken from
- neighborIP: 10.77.33.3 # "MultiRackCluster.spec.defaultPeer" if
# not set here
As compared to single rack examples, the following Machine
objects differ
in:
BMH selectors
L2Template selectors
Rack selectors (the
ipam/RackRef
label)The
rack-id
node labelsThe labels on master nodes are required for MetalLB node selectors if MetalLB is used to announce LB IP addresses on master nodes. In this scenario, the L2 (ARP) announcement mode cannot be used for MetalLB because master nodes are in different L2 segments. So, the BGP announcement mode must be used for MetalLB. Node selectors are required to properly configure BGP connections from each master node.
Configuration example for Machine
1
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
name: test-cluster-master-1
namespace: managed-ns
annotations:
metal3.io/BareMetalHost: managed-ns/test-cluster-master-1
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
cluster.sigs.k8s.io/control-plane: controlplane
hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
ipam/RackRef: rack-master-1
kaas.mirantis.com/provider: baremetal
spec:
providerSpec:
value:
kind: BareMetalMachineProviderSpec
apiVersion: baremetal.k8s.io/v1alpha1
hostSelector:
matchLabels:
kaas.mirantis.com/baremetalhost-id: test-cluster-master-1
l2TemplateSelector:
name: test-cluster-master-1
nodeLabels: # not used for BGP announcement of the
- key: rack-id # cluster API LB IP but can be used for
value: rack-master-1 # MetalLB if "nodeSelectors" are required
Configuration example for Machine
2
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
name: test-cluster-master-2
namespace: managed-ns
annotations:
metal3.io/BareMetalHost: managed-ns/test-cluster-master-2
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
cluster.sigs.k8s.io/control-plane: controlplane
hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
ipam/RackRef: rack-master-2
kaas.mirantis.com/provider: baremetal
spec:
providerSpec:
value:
kind: BareMetalMachineProviderSpec
apiVersion: baremetal.k8s.io/v1alpha1
hostSelector:
matchLabels:
kaas.mirantis.com/baremetalhost-id: test-cluster-master-2
l2TemplateSelector:
name: test-cluster-master-2
nodeLabels: # not used for BGP announcement of the
- key: rack-id # cluster API LB IP but can be used for
value: rack-master-2 # MetalLB if "nodeSelectors" are required
Configuration example for Machine
3
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
name: test-cluster-master-3
namespace: managed-ns
annotations:
metal3.io/BareMetalHost: managed-ns/test-cluster-master-3
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
cluster.sigs.k8s.io/control-plane: controlplane
hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
ipam/RackRef: rack-master-3
kaas.mirantis.com/provider: baremetal
spec:
providerSpec:
value:
kind: BareMetalMachineProviderSpec
apiVersion: baremetal.k8s.io/v1alpha1
hostSelector:
matchLabels:
kaas.mirantis.com/baremetalhost-id: test-cluster-master-3
l2TemplateSelector:
name: test-cluster-master-3
nodeLabels: # optional. not used for BGP announcement of
- key: rack-id # the cluster API LB IP but can be used for
value: rack-master-3 # MetalLB if "nodeSelectors" are required
Configuration example for Subnet
defining the cluster API
LB IP address
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
name: test-cluster-api-lb
namespace: managed-ns
labels:
kaas.mirantis.com/provider: baremetal
ipam/SVC-LBhost: "1"
cluster.sigs.k8s.io/cluster-name: test-cluster
spec:
cidr: 134.33.24.201/32
useWholeCidr: true
Configuration example for Subnet
of the LCM network in the
rack-master-1
rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: lcm-rack-control-1
namespace: managed-ns
spec:
cidr: 10.77.31.0/28
gateway: 10.77.31.1
includeRanges:
- 10.77.31.4-10.77.31.13
nameservers:
- 1.2.3.4
Configuration example for Subnet
of the LCM network in the
rack-master-2
rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: lcm-rack-control-2
namespace: managed-ns
spec:
cidr: 10.77.32.0/28
gateway: 10.77.32.1
includeRanges:
- 10.77.32.4-10.77.32.13
nameservers:
- 1.2.3.4
Configuration example for Subnet
of the LCM network in the
rack-master-3
rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: lcm-rack-control-3
namespace: managed-ns
spec:
cidr: 10.77.33.0/28
gateway: 10.77.33.1
includeRanges:
- 10.77.33.4-10.77.33.13
nameservers:
- 1.2.3.4
The following L2Template
objects differ in LCM and external subnets that
each master node uses.
Configuration example for L2Template
1
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: test-cluster-master-1
namespace: managed-ns
spec:
...
l3Layout:
- subnetName: lcm-rack-control-1 # this network is referenced
scope: namespace # in the "rack-master-1" Rack
- subnetName: ext-rack-control-1 # this optional network is used for
scope: namespace # Kubernetes services traffic and
# MetalLB BGP connections
...
npTemplate: |
...
ethernets:
lo:
addresses:
- {{ cluster_api_lb_ip }} # function for cluster API LB IP
dhcp4: false
dhcp6: false
...
Configuration example for L2Template
2
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: test-cluster-master-2
namespace: managed-ns
spec:
...
l3Layout:
- subnetName: lcm-rack-control-2 # this network is referenced
scope: namespace # in "rack-master-2" Rack
- subnetName: ext-rack-control-2 # this network is used for Kubernetes services
scope: namespace # traffic and MetalLB BGP connections
...
npTemplate: |
...
ethernets:
lo:
addresses:
- {{ cluster_api_lb_ip }} # function for cluster API LB IP
dhcp4: false
dhcp6: false
...
Configuration example for L2Template
3
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: test-cluster-master-3
namespace: managed-ns
spec:
...
l3Layout:
- subnetName: lcm-rack-control-3 # this network is referenced
scope: namespace # in "rack-master-3" Rack
- subnetName: ext-rack-control-3 # this network is used for Kubernetes services
scope: namespace # traffic and MetalLB BGP connections
...
npTemplate: |
...
ethernets:
lo:
addresses:
- {{ cluster_api_lb_ip }} # function for cluster API LB IP
dhcp4: false
dhcp6: false
...
The following MetalLBConfig
example illustrates how node labels
are used in nodeSelectors
of bgpPeers
. Each of bgpPeers
corresponds to one of master nodes.
Configuration example for MetalLBConfig
apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
labels:
cluster.sigs.k8s.io/cluster-name: test-cluster
kaas.mirantis.com/provider: baremetal
name: test-cluster-metallb-config
namespace: managed-ns
spec:
...
bgpPeers:
- name: svc-peer-rack1
spec:
holdTime: 0s
keepaliveTime: 0s
peerAddress: 10.77.41.1 # peer address is in external subnet
# instead of LCM subnet used for BGP
# connection to announce cluster API LB IP
peerASN: 65100 # the same as for BGP connection used to announce
# cluster API LB IP
myASN: 65101 # the same as for BGP connection used to announce
# cluster API LB IP
nodeSelectors:
- matchLabels:
rack-id: rack-master-1 # references the node corresponding
# to "test-cluster-master-1" Machine
- name: svc-peer-rack2
spec:
holdTime: 0s
keepaliveTime: 0s
peerAddress: 10.77.42.1
peerASN: 65100
myASN: 65101
nodeSelectors:
- matchLabels:
rack-id: rack-master-1
- name: svc-peer-rack3
spec:
holdTime: 0s
keepaliveTime: 0s
peerAddress: 10.77.43.1
peerASN: 65100
myASN: 65101
nodeSelectors:
- matchLabels:
rack-id: rack-master-1
...
After the objects are created and nodes are provisioned, the IpamHost
objects will have BGP daemon configuration files in their status
fields.
Refer to Single rack configuration example on how to verify the BGP configuration
files.