Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Starting with MOSK 25.2, the MOSK documentation set covers all product layers, including MOSK management (formerly Container Cloud). This means everything you need is in one place. Some legacy names may remain in the code and documentation and will be updated in future releases. The separate Container Cloud documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

MultiRackCluster

TechPreview Available since MCC 2.24.4 (15.0.3 and 14.0.3)

This section describes the MultiRackCluster resource used in the management API.

When you create a cluster with a multi-rack topology, where Kubernetes masters are distributed across multiple racks without L2 layer extension between them, the MultiRackCluster resource allows you to set cluster-wide parameters for configuration of the BGP announcement of the cluster API load balancer address. In this scenario, the MultiRackCluster object must be bound to the Cluster object.

The MultiRackCluster object is generally used for a particular cluster in conjunction with Rack objects described in Rack.

For demonstration purposes, we split the MultiRackCluster custom resource (CR) description into the following major sections:

MultiRackCluster metadata

The MultiRackCluster CR metadata contains the following fields:

  • apiVersion

    API version of the object that is ipam.mirantis.com/v1alpha1.

  • kind

    Object type that is MultiRackCluster.

  • metadata

    The metadata field contains the following subfields:

    • name

      Name of the MultiRackCluster object.

    • namespace

      MOSK project (Kubernetes namespace) in which the object was created.

    • labels

      Key-value pairs that are attached to the object:

      • cluster.sigs.k8s.io/cluster-name

        Cluster object name that this MultiRackCluster object is applied to. To enable the use of BGP announcement for the cluster API LB address, set the useBGPAnnouncement parameter in the Cluster object to true:

        spec:
  providerSpec:
    value:
      useBGPAnnouncement: true
      • kaas.mirantis.com/provider

        Provider name that is baremetal.

      • kaas.mirantis.com/region

        Region name.

        Note

        The kaas.mirantis.com/region label is removed from all MOSK objects in 24.1. Therefore, do not add the label starting with this release. On existing clusters updated to this release, or if added manually, MOSK ignores this label.

      Warning

      Labels and annotations that are not documented in this API Reference are generated automatically. Do not modify them using the API.

The MultiRackCluster metadata configuration example:

apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
  name: multirack-test-cluster
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal

MultiRackCluster spec

The spec field of the MultiRackCluster resource describes the desired state of the object. It contains the following fields:

  • bgpdConfigFileName

    Name of the configuration file for the BGP daemon (bird). Recommended value is bird.conf.

  • bgpdConfigFilePath

    Path to the directory where the configuration file for the BGP daemon (bird) is added. The recommended value is /etc/bird.

  • bgpdConfigTemplate

    Optional. Configuration text file template for the BGP daemon (bird) configuration file where you can use go template constructs and the following variables:

    • RouterID, LocalIP

      Local IP on the given network, which is a key in the Rack.spec.peeringMap dictionary, for a given node. You can use it, for example, in the router id {{$.RouterID}}; instruction.

    • LocalASN

      Local AS number.

    • NeighborASN

      Neighbor AS number.

    • NeighborIP

      Neighbor IP address. Its values are taken from Rack.spec.peeringMap, it can be used only inside the range iteration through the Neighbors list.

    • Neighbors

      List of peers in the given network and node. It can be iterated through the range statement in the go template.

    Values for LocalASN and NeighborASN are taken from:

    • MultiRackCluster.defaultPeer - if not used as a field inside the range iteration through the Neighbors list.

    • Corresponding values of Rack.spec.peeringMap - if used as a field inside the range iteration through the Neighbors list.

    This template can be overridden using the Rack objects. For details, see Rack spec.

  • defaultPeer

    Configuration parameters for the default BGP peer. These parameters will be used in rendering of the configuration file for BGP daemon from the template if they are not overridden for a particular rack or network using Rack objects. For details, see Rack spec.

    • localASN

      Mandatory. Local AS number.

    • neighborASN

      Mandatory. Neighbor AS number.

    • neighborIP

      Reserved. Neighbor IP address. Leave it as an empty string.

    • password

      Optional. Neighbor password. If not set, you can hardcode it in bgpdConfigTemplate. It is required for MD5 authentication between BGP peers.

Configuration examples:

Since MCC 2.26.0 (17.1.0 and 16.1.0) for bird v2.x
spec:
  bgpdConfigFileName: bird.conf
  bgpdConfigFilePath: /etc/bird
  bgpdConfigTemplate: |
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
      ipv4;
    }
    #
    protocol kernel {
      ipv4 {
        export all;
      };
    }
    #
    {{range $i, $peer := .Neighbors}}
    protocol bgp 'bgp_peer_{{$i}}' {
      local port 1179 as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      ipv4 {
        import none;
        export filter {
          if dest = RTD_UNREACHABLE then {
            reject;
          }
          accept;
        };
      };
    }
    {{end}}
  defaultPeer:
    localASN: 65101
    neighborASN: 65100
    neighborIP: ""
Before MCC 2.26.0 (17.0.0, 16.0.0, or earlier) for bird v1.x
spec:
  bgpdConfigFileName: bird.conf
  bgpdConfigFilePath: /etc/bird
  bgpdConfigTemplate: |
    listen bgp port 1179;
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
    }
    #
    protocol kernel {
      export all;
    }
    #
    {{range $i, $peer := .Neighbors}}
    protocol bgp 'bgp_peer_{{$i}}' {
      local as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      import all;
      export filter {
        if dest = RTD_UNREACHABLE then {
          reject;
        }
        accept;
      };
    }
    {{end}}
  defaultPeer:
    localASN: 65101
    neighborASN: 65100
    neighborIP: ""

MultiRackCluster status

The status field of the MultiRackCluster resource reflects the actual state of the MultiRackCluster object and contains the following fields:

  • state Since MCC 2.23.0 (11.7.0)

    Message that reflects the current status of the resource. The list of possible values includes the following:

    • OK - object is operational.

    • ERR - object is non-operational. This status has a detailed description in the messages list.

    • TERM - object was deleted and is terminating.

  • messages Since MCC 2.23.0 (11.7.0)

    List of error or warning messages if the object state is ERR.

  • objCreated

    Date, time, and IPAM version of the resource creation.

  • objStatusUpdated

    Date, time, and IPAM version of the last update of the status field in the resource.

  • objUpdated

    Date, time, and IPAM version of the last resource update.

Configuration example:

status:
  checksums:
    annotations: sha256:38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed
    labels: sha256:d8f8eacf487d57c22ca0ace29bd156c66941a373b5e707d671dc151959a64ce7
    spec: sha256:66b5d28215bdd36723fe6230359977fbede828906c6ae96b5129a972f1fa51e9
  objCreated: 2023-08-11T12:25:21.00000Z  by  v6.5.999-20230810-155553-2497818
  objStatusUpdated: 2023-08-11T12:32:58.11966Z  by  v6.5.999-20230810-155553-2497818
  objUpdated: 2023-08-11T12:32:57.32036Z  by  v6.5.999-20230810-155553-2497818
  state: OK

MultiRackCluster and Rack usage examples

The following configuration examples of several bare metal objects illustrate how to configure BGP announcement of the load balancer address used to expose the cluster API.

Single rack configuration example

In the following example, all master nodes are in a single rack. One Rack object is required in this case for master nodes. Some worker nodes can coexist in the same rack with master nodes or occupy separate racks. It is implied that the useBGPAnnouncement parameter is set to true in the corresponding Cluster object.

Configuration example for MultiRackCluster

Since MCC 2.26.0 (Cluster releases 17.1.0 and 16.1.0) bird v2.x:

apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
  name: multirack-test-cluster
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  bgpdConfigFileName: bird.conf
  bgpdConfigFilePath: /etc/bird
  bgpdConfigTemplate: |
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
      ipv4;
    }
    #
    protocol kernel {
      ipv4 {
        export all;
      };
    }
    #
    {{range $i, $peer := .Neighbors}}
    protocol bgp 'bgp_peer_{{$i}}' {
      local port 1179 as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      ipv4 {
        import none;
        export filter {
          if dest = RTD_UNREACHABLE then {
            reject;
          }
          accept;
        };
      };
    }
    {{end}}
  defaultPeer:
    localASN: 65101
    neighborASN: 65100
    neighborIP: ""

Before MCC 2.26.0 (Cluster releases 17.0.0, 16.0.0, or earlier) bird v1.x:

apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
  name: multirack-test-cluster
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  bgpdConfigFileName: bird.conf
  bgpdConfigFilePath: /etc/bird
  bgpdConfigTemplate: |
    listen bgp port 1179;
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
    }
    #
    protocol kernel {
      export all;
    }
    #
    {{range $i, $peer := .Neighbors}}
    protocol bgp 'bgp_peer_{{$i}}' {
      local as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      import all;
      export filter {
        if dest = RTD_UNREACHABLE then {
          reject;
        }
        accept;
      };
    }
    {{end}}
  defaultPeer:
    localASN: 65101
    neighborASN: 65100
    neighborIP: ""
Configuration example for Rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
  name: rack-master
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  peeringMap:
    lcm-rack-control:
      peers:
      - neighborIP: 10.77.31.1  # "localASN" and "neighborASN" are taken from
      - neighborIP: 10.77.37.1  # "MultiRackCluster.spec.defaultPeer"
                                # if not set here
Configuration example for Machine
# "Machine" templates for "test-cluster-master-2" and "test-cluster-master-3"
# differ only in BMH selectors in this example.
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-1
  namespace: test-ns
  annotations:
    metal3.io/BareMetalHost: test-ns/test-cluster-master-1
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master # used to connect "IpamHost" to "Rack" objects, so that
                              # BGP parameters can be obtained from "Rack" to
                              # render BGP configuration for the given "IpamHost" object
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-1
      l2TemplateSelector:
        name: test-cluster-master

Note

Before update of the management cluster to 2.29.0 (Cluster release 16.4.0), instead of BareMetalHostInventory, use the BareMetalHost object. For details, see BareMetalHost resource.

Caution

While the Cluster release of the management cluster is 16.4.0, BareMetalHostInventory operations are allowed to m:kaas@management-admin only. This limitation is lifted once the management cluster is updated to the Cluster release 16.4.1 or later.

Configuration example for L2Template
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: test-cluster-master
  namespace: test-ns
spec:
  ...
  l3Layout:
    - subnetName: lcm-rack-control # this network is referenced in "rack-master" Rack
      scope:      namespace
  ...
  npTemplate: |
    ...
    ethernets:
      lo:
        addresses:
          - {{ cluster_api_lb_ip }}  # function for cluster API LB IP
        dhcp4: false
        dhcp6: false
    ...

After the objects are created and nodes are provisioned, the IpamHost objects will have BGP daemon configuration files in their status fields. For example:

Configuration example for IpamHost
apiVersion: ipam.mirantis.com/v1alpha1
kind: IpamHost
...
status:
  ...
  netconfigFiles:
  - content: bGlzdGVuIGJncCBwb3J0IDExNzk7CnByb3RvY29sIGRldmljZSB7Cn0KIwpwcm90b2NvbCBkaXJlY3QgewogIGludGVyZmFjZSAibG8iOwp9CiMKcHJvdG9jb2wga2VybmVsIHsKICBleHBvcnQgYWxsOwp9CiMKCnByb3RvY29sIGJncCAnYmdwX3BlZXJfMCcgewogIGxvY2FsIGFzIDY1MTAxOwogIG5laWdoYm9yIDEwLjc3LjMxLjEgYXMgNjUxMDA7CiAgaW1wb3J0IGFsbDsKICBleHBvcnQgZmlsdGVyIHsKICAgIGlmIGRlc3QgPSBSVERfVU5SRUFDSEFCTEUgdGhlbiB7CiAgICAgIHJlamVjdDsKICAgIH0KICAgIGFjY2VwdDsKICB9Owp9Cgpwcm90b2NvbCBiZ3AgJ2JncF9wZWVyXzEnIHsKICBsb2NhbCBhcyA2NTEwMTsKICBuZWlnaGJvciAxMC43Ny4zNy4xIGFzIDY1MTAwOwogIGltcG9ydCBhbGw7CiAgZXhwb3J0IGZpbHRlciB7CiAgICBpZiBkZXN0ID0gUlREX1VOUkVBQ0hBQkxFIHRoZW4gewogICAgICByZWplY3Q7CiAgICB9CiAgICBhY2NlcHQ7CiAgfTsKfQoK
    path: /etc/bird/bird.conf
  - content: ...
    path: /etc/netplan/60-kaas-lcm-netplan.yaml
  netconfigFilesStates:
    /etc/bird/bird.conf: 'OK: 2023-08-17T08:00:58.96140Z 25cde040e898fd5bf5b28aacb12f046b4adb510570ecf7d7fa5a8467fa4724ec'
    /etc/netplan/60-kaas-lcm-netplan.yaml: 'OK: 2023-08-11T12:33:24.54439Z 37ac6e9fe13e5969f35c20c615d96b4ed156341c25e410e95831794128601e01'
  ...

You can decode /etc/bird/bird.conf contents and verify the configuration:

echo "<<base64-string>>" | base64 -d

The following system output applies to the above configuration examples:

Configuration example for the decoded bird.conf

Since MCC 2.26.0 (Cluster releases 17.1.0 and 16.1.0) bird v2.x:

protocol device {
}
#
protocol direct {
  interface "lo";
  ipv4;
}
#
protocol kernel {
  ipv4 {
    export all;
  };
}
#

protocol bgp 'bgp_peer_0' {
  local port 1179 as 65101;
  neighbor 10.77.31.1 as 65100;
  ipv4 {
    import none;
    export filter {
      if dest = RTD_UNREACHABLE then {
        reject;
      }
      accept;
    };
  };
}

protocol bgp 'bgp_peer_1' {
  local port 1179 as 65101;
  neighbor 10.77.37.1 as 65100;
  ipv4 {
    import none;
    export filter {
      if dest = RTD_UNREACHABLE then {
        reject;
      }
      accept;
    };
  };
}

Before MCC 2.26.0 (Cluster releases 17.0.0, 16.0.0, or earlier) bird v1.x:

listen bgp port 1179;
protocol device {
}
#
protocol direct {
  interface "lo";
}
#
protocol kernel {
  export all;
}
#

protocol bgp 'bgp_peer_0' {
  local as 65101;
  neighbor 10.77.31.1 as 65100;
  import all;
  export filter {
    if dest = RTD_UNREACHABLE then {
      reject;
    }
    accept;
  };
}

protocol bgp 'bgp_peer_1' {
  local as 65101;
  neighbor 10.77.37.1 as 65100;
  import all;
  export filter {
    if dest = RTD_UNREACHABLE then {
      reject;
    }
    accept;
  };
}

BGP daemon configuration files are copied from IpamHost.status to the corresponding LCMMachine object the same way as it is done for netplan configuration files. Then, the configuration files are written to the corresponding node by the LCM-Agent.

Multiple rack configuration example

In the following configuration example, each master node is located in its own rack. Three Rack objects are required in this case for master nodes. Some worker nodes can coexist in the same racks with master nodes or occupy separate racks. Only objects that are required to show configuration for BGP announcement of the cluster API load balancer address are provided here.

For the description of Rack, MetalLBConfig, and other objects that are required for MetalLB configuration in this scenario, refer to Configuration example for using BGP announcement.

It is implied that the useBGPAnnouncement parameter is set to true in the corresponding Cluster object.

Configuration example for MultiRackCluster

Since MCC 2.26.0 (Cluster releases 17.1.0 and 16.1.0) bird v2.x:

# It is the same object as in the single rack example.
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
  name: multirack-test-cluster
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  bgpdConfigFileName: bird.conf
  bgpdConfigFilePath: /etc/bird
  bgpdConfigTemplate: |
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
      ipv4;
    }
    #
    protocol kernel {
      ipv4 {
        export all;
      };
    }
    #
    {{range $i, $peer := .Neighbors}}
    protocol bgp 'bgp_peer_{{$i}}' {
      local port 1179 as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      ipv4 {
        import none;
        export filter {
          if dest = RTD_UNREACHABLE then {
            reject;
          }
          accept;
        };
      };
    }
    {{end}}
  defaultPeer:
    localASN: 65101
    neighborASN: 65100
    neighborIP: ""

Before MCC 2.26.0 (Cluster releases 17.0.0, 16.0.0, or earlier) bird v1.x:

# It is the same object as in the single rack example.
apiVersion: ipam.mirantis.com/v1alpha1
kind: MultiRackCluster
metadata:
  name: multirack-test-cluster
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  bgpdConfigFileName: bird.conf
  bgpdConfigFilePath: /etc/bird
  bgpdConfigTemplate: |
    listen bgp port 1179;
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
    }
    #
    protocol kernel {
      export all;
    }
    #
    {{range $i, $peer := .Neighbors}}
    protocol bgp 'bgp_peer_{{$i}}' {
      local as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      import all;
      export filter {
        if dest = RTD_UNREACHABLE then {
          reject;
        }
        accept;
      };
    }
    {{end}}
  defaultPeer:
    localASN: 65101
    neighborASN: 65100
    neighborIP: ""

The following Rack objects differ in neighbor IP addresses and in the network (L3 subnet) used for BGP connection to announce the cluster API LB IP and for cluster API traffic.

Configuration example for Rack 1
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
  name: rack-master-1
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  peeringMap:
    lcm-rack-control-1:
      peers:
      - neighborIP: 10.77.31.2  # "localASN" and "neighborASN" are taken from
      - neighborIP: 10.77.31.3  # "MultiRackCluster.spec.defaultPeer" if
                                # not set here
Configuration example for Rack 2
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
  name: rack-master-2
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  peeringMap:
    lcm-rack-control-2:
      peers:
      - neighborIP: 10.77.32.2  # "localASN" and "neighborASN" are taken from
      - neighborIP: 10.77.32.3  # "MultiRackCluster.spec.defaultPeer" if
                                # not set here
Configuration example for Rack 3
apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
  name: rack-master-3
  namespace: test-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  peeringMap:
    lcm-rack-control-3:
      peers:
      - neighborIP: 10.77.33.2  # "localASN" and "neighborASN" are taken from
      - neighborIP: 10.77.33.3  # "MultiRackCluster.spec.defaultPeer" if
                                # not set here

As compared to single rack examples, the following Machine objects differ in:

  • BMH selectors

  • L2Template selectors

  • Rack selectors (the ipam/RackRef label)

  • The rack-id node labels

    The labels on master nodes are required for MetalLB node selectors if MetalLB is used to announce LB IP addresses on master nodes. In this scenario, the L2 (ARP) announcement mode cannot be used for MetalLB because master nodes are in different L2 segments. So, the BGP announcement mode must be used for MetalLB. Node selectors are required to properly configure BGP connections from each master node.

Note

Before update of the management cluster to 2.29.0 (Cluster release 16.4.0), instead of BareMetalHostInventory, use the BareMetalHost object. For details, see BareMetalHost resource.

Caution

While the Cluster release of the management cluster is 16.4.0, BareMetalHostInventory operations are allowed to m:kaas@management-admin only. This limitation is lifted once the management cluster is updated to the Cluster release 16.4.1 or later.

Configuration example for Machine 1
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-1
  namespace: test-ns
  annotations:
    metal3.io/BareMetalHost: test-ns/test-cluster-master-1
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master-1
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-1
      l2TemplateSelector:
        name: test-cluster-master-1
      nodeLabels:             # not used for BGP announcement of the
      - key: rack-id          # cluster API LB IP but can be used for
        value: rack-master-1  # MetalLB if "nodeSelectors" are required
Configuration example for Machine 2
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-2
  namespace: test-ns
  annotations:
    metal3.io/BareMetalHost: test-ns/test-cluster-master-2
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master-2
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-2
      l2TemplateSelector:
        name: test-cluster-master-2
      nodeLabels:             # not used for BGP announcement of the
      - key: rack-id          # cluster API LB IP but can be used for
        value: rack-master-2  # MetalLB if "nodeSelectors" are required
Configuration example for Machine 3
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-3
  namespace: test-ns
  annotations:
    metal3.io/BareMetalHost: test-ns/test-cluster-master-3
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master-3
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-3
      l2TemplateSelector:
        name: test-cluster-master-3
      nodeLabels:             # optional. not used for BGP announcement of
      - key: rack-id          # the cluster API LB IP but can be used for
        value: rack-master-3  # MetalLB if "nodeSelectors" are required
Configuration example for Subnet defining the cluster API LB IP address
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: test-cluster-api-lb
  namespace: test-ns
  labels:
    kaas.mirantis.com/provider: baremetal
    ipam/SVC-LBhost: "1"
    cluster.sigs.k8s.io/cluster-name: test-cluster
spec:
  cidr: 134.33.24.201/32
  useWholeCidr: true
Configuration example for Subnet of the LCM network in the rack-master-1 rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: lcm-rack-control-1
  namespace: test-ns
spec:
  cidr: 10.77.31.0/28
  gateway: 10.77.31.1
  includeRanges:
    - 10.77.31.4-10.77.31.13
  nameservers:
    - 1.2.3.4
Configuration example for Subnet of the LCM network in the rack-master-2 rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: lcm-rack-control-2
  namespace: test-ns
spec:
  cidr: 10.77.32.0/28
  gateway: 10.77.32.1
  includeRanges:
    - 10.77.32.4-10.77.32.13
  nameservers:
    - 1.2.3.4
Configuration example for Subnet of the LCM network in the rack-master-3 rack
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: lcm-rack-control-3
  namespace: test-ns
spec:
  cidr: 10.77.33.0/28
  gateway: 10.77.33.1
  includeRanges:
    - 10.77.33.4-10.77.33.13
  nameservers:
    - 1.2.3.4

The following L2Template objects differ in LCM and external subnets that each master node uses.

Configuration example for L2Template 1
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: test-cluster-master-1
  namespace: test-ns
spec:
  ...
  l3Layout:
    - subnetName: lcm-rack-control-1  # this network is referenced
      scope:      namespace           # in the "rack-master-1" Rack
    - subnetName: ext-rack-control-1  # this optional network is used for
      scope:      namespace           # Kubernetes services traffic and
                                      # MetalLB BGP connections
  ...
  npTemplate: |
    ...
    ethernets:
      lo:
        addresses:
          - {{ cluster_api_lb_ip }}  # function for cluster API LB IP
        dhcp4: false
        dhcp6: false
    ...
Configuration example for L2Template 2
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: test-cluster-master-2
  namespace: test-ns
spec:
  ...
  l3Layout:
    - subnetName: lcm-rack-control-2  # this network is referenced
      scope:      namespace           # in "rack-master-2" Rack
    - subnetName: ext-rack-control-2  # this network is used for Kubernetes services
      scope:      namespace           # traffic and MetalLB BGP connections
  ...
  npTemplate: |
    ...
    ethernets:
      lo:
        addresses:
          - {{ cluster_api_lb_ip }}  # function for cluster API LB IP
        dhcp4: false
        dhcp6: false
    ...
Configuration example for L2Template 3
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: test-cluster-master-3
  namespace: test-ns
spec:
  ...
  l3Layout:
    - subnetName: lcm-rack-control-3  # this network is referenced
      scope:      namespace           # in "rack-master-3" Rack
    - subnetName: ext-rack-control-3  # this network is used for Kubernetes services
      scope:      namespace           # traffic and MetalLB BGP connections
  ...
  npTemplate: |
    ...
    ethernets:
      lo:
        addresses:
          - {{ cluster_api_lb_ip }}  # function for cluster API LB IP
        dhcp4: false
        dhcp6: false
    ...

The following MetalLBConfig example illustrates how node labels are used in nodeSelectors of bgpPeers. Each of bgpPeers corresponds to one of master nodes.

Configuration example for MetalLBConfig
apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
labels:
  cluster.sigs.k8s.io/cluster-name: test-cluster
  kaas.mirantis.com/provider: baremetal
name: test-cluster-metallb-config
namespace: test-ns
spec:
  ...
  bgpPeers:
    - name: svc-peer-rack1
      spec:
        holdTime: 0s
        keepaliveTime: 0s
        peerAddress: 10.77.41.1 # peer address is in external subnet
                                # instead of LCM subnet used for BGP
                                # connection to announce cluster API LB IP
        peerASN: 65100  # the same as for BGP connection used to announce
                        # cluster API LB IP
        myASN: 65101    # the same as for BGP connection used to announce
                        # cluster API LB IP
        nodeSelectors:
          - matchLabels:
              rack-id: rack-master-1  # references the node corresponding
                                      # to "test-cluster-master-1" Machine
    - name: svc-peer-rack2
      spec:
        holdTime: 0s
        keepaliveTime: 0s
        peerAddress: 10.77.42.1
        peerASN: 65100
        myASN: 65101
        nodeSelectors:
          - matchLabels:
              rack-id: rack-master-1
    - name: svc-peer-rack3
      spec:
        holdTime: 0s
        keepaliveTime: 0s
        peerAddress: 10.77.43.1
        peerASN: 65100
        myASN: 65101
        nodeSelectors:
          - matchLabels:
              rack-id: rack-master-1
  ...

After the objects are created and nodes are provisioned, the IpamHost objects will have BGP daemon configuration files in their status fields. Refer to Single rack configuration example on how to verify the BGP configuration files.