Rack

TechPreview Available since 2.24.4

This section describes the Rack resource used in the Container Cloud API.

When you create a bare metal managed cluster with a multi-rack topology, where Kubernetes masters are distributed across multiple racks without L2 layer extension between them, the Rack resource allows you to configure BGP announcement of the cluster API load balancer address from each rack.

In this scenario, Rack objects must be bound to Machine objects corresponding to master nodes of the cluster. Each Rack object describes the configuration of the BGP daemon (bird) used to announce the cluster API LB address from a particular master node (or from several nodes in the same rack).

Rack objects are used for a particular cluster only in conjunction with the MultiRackCluster object described in MultiRackCluster.

For demonstration purposes, the Container Cloud Rack custom resource (CR) description is split into the following major sections:

For configuration examples, see MultiRackCluster and Rack usage examples.

Rack metadata

The Container Cloud Rack CR metadata contains the following fields:

  • apiVersion

    API version of the object that is ipam.mirantis.com/v1alpha1.

  • kind

    Object type that is Rack.

  • metadata

    The metadata field contains the following subfields:

    • name

      Name of the Rack object. Corresponding Machine objects must have their ipam/RackRef label value set to the name of the Rack object. This label is required only for Machine objects of the master nodes that announce the cluster API LB address.

    • namespace

      Container Cloud project (Kubernetes namespace) where the object was created.

    • labels

      Key-value pairs that are attached to the object:

      • cluster.sigs.k8s.io/cluster-name

        Cluster object name that this Rack object is applied to.

      • kaas.mirantis.com/provider

        Provider name that is baremetal.

      • kaas.mirantis.com/region

        Region name.

        Note

        The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.

      Warning

      Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.

Rack metadata example:

apiVersion: ipam.mirantis.com/v1alpha1
kind: Rack
metadata:
  name: rack-1
  namespace: managed-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal

Corresponding Machine metadata example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-1
    kaas.mirantis.com/provider: baremetal
  name: managed-master-1-control-efi-6tg52
  namespace: managed-ns

Rack spec

The spec field of the Rack resource describes the desired state of the object. It contains the following fields:

  • bgpdConfigTemplate

    Optional. Configuration file template that will be used to create configuration file for a BGP daemon on nodes in this rack. If not set, the configuration file template from the corresponding MultiRackCluster object is used.

  • peeringMap

    Structure that describes general parameters of BGP peers to be used in the configuration file for a BGP daemon for each network where BGP announcement is used. Also, you can define a separate configuration file template for the BGP daemon for each of those networks. The peeringMap structure is as follows:

    peeringMap:
  <network-name-a>:
    peers:
      - localASN: <localASN-1>
        neighborASN: <neighborASN-1>
        neighborIP: <neighborIP-1>
        password: <password-1>
      - localASN: <localASN-2>
        neighborASN: <neighborASN-2>
        neighborIP: <neighborIP-2>
        password: <password-2>
    bgpdConfigTemplate: |
    <configuration file template for a BGP daemon>
  ...
    • <network-name-a>

      Name of the network where a BGP daemon should connect to the neighbor BGP peers. By default, it is implied that the same network is used on the node to make connection to the neighbor BGP peers as well as to receive and respond to the traffic directed to the IP address being advertised. In our scenario, the advertised IP address is the cluster API LB IP address.

      This network name must be the same as the subnet name used in the L2 template (l3Layout section) for the corresponding master node(s).

    • peers

      Optional. List of dictionaries where each dictionary defines configuration parameters for a particular BGP peer. Peer parameters are as follows:

      • localASN

        Optional. Local AS number. If not set, it can be taken from MultiRackCluster.spec.defaultPeer or can be hardcoded in bgpdConfigTemplate.

      • neighborASN

        Optional. Neighbor AS number. If not set, it can be taken from MultiRackCluster.spec.defaultPeer or can be hardcoded in bgpdConfigTemplate.

      • neighborIP

        Mandatory. Neighbor IP address.

      • password

        Optional. Neighbor password. If not set, it can be taken from MultiRackCluster.spec.defaultPeer or can be hardcoded in bgpdConfigTemplate. It is required when MD5 authentication between BGP peers is used.

    • bgpdConfigTemplate

      Optional. Configuration file template that will be used to create the configuration file for the BGP daemon of the network-name-a network on a particular node. If not set, Rack.spec.bgpdConfigTemplate is used.

Configuration example:

Since Cluster releases 17.1.0 and 16.1.0 for bird v2.x
spec:
  bgpdConfigTemplate: |
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
      ipv4;
    }
    #
    protocol kernel {
      ipv4 {
        export all;
      };
    }
    #
    protocol bgp bgp_lcm {
      local port 1179 as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      ipv4 {
         import none;
         export filter {
           if dest = RTD_UNREACHABLE then {
             reject;
           }
           accept;
         };
      };
    }
  peeringMap:
    lcm-rack1:
      peers:
      - localASN: 65050
        neighborASN: 65011
        neighborIP: 10.77.31.1
Before Cluster releases 17.1.0 and 16.1.0 for bird v1.x
spec:
  bgpdConfigTemplate: |
    listen bgp port 1179;
    protocol device {
    }
    #
    protocol direct {
      interface "lo";
    }
    #
    protocol kernel {
      export all;
    }
    #
    protocol bgp bgp_lcm {
      local as {{.LocalASN}};
      neighbor {{.NeighborIP}} as {{.NeighborASN}};
      import all;
      export filter {
        if dest = RTD_UNREACHABLE then {
          reject;
        }
        accept;
      };
    }
  peeringMap:
    lcm-rack1:
      peers:
      - localASN: 65050
        neighborASN: 65011
        neighborIP: 10.77.31.1

Rack status

The status field of the Rack resource reflects the actual state of the Rack object and contains the following fields:

  • state Since 2.23.0

    Message that reflects the current status of the resource. The list of possible values includes the following:

    • OK - object is operational.

    • ERR - object is non-operational. This status has a detailed description in the messages list.

    • TERM - object was deleted and is terminating.

  • messages Since 2.23.0

    List of error or warning messages if the object state is ERR.

  • objCreated

    Date, time, and IPAM version of the resource creation.

  • objStatusUpdated

    Date, time, and IPAM version of the last update of the status field in the resource.

  • objUpdated

    Date, time, and IPAM version of the last resource update.

Configuration example:

status:
  checksums:
    annotations: sha256:cd4b751d9773eacbfd5493712db0cbebd6df0762156aefa502d65a9d5e8af31d
    labels: sha256:fc2612d12253443955e1bf929f437245d304b483974ff02a165bc5c78363f739
    spec: sha256:8f0223b1eefb6a9cd583905a25822fd83ac544e62e1dfef26ee798834ef4c0c1
  objCreated: 2023-08-11T12:25:21.00000Z  by  v6.5.999-20230810-155553-2497818
  objStatusUpdated: 2023-08-11T12:33:00.92163Z  by  v6.5.999-20230810-155553-2497818
  objUpdated: 2023-08-11T12:32:59.11951Z  by  v6.5.999-20230810-155553-2497818
  state: OK