MetalLBConfigTemplate

Deprecated in 2.27.0 (17.2.0 and 16.2.0)

Warning

The MetalLBConfigTemplate object may not work as expected due to its deprecation. Existing MetalLBConfigTemplate objects and related Subnet objects will be automatically migrated to MetallbConfig during cluster update to the Cluster release 17.2.0 or 16.2.0.

Support status of MetalLBConfigTemplate

Container Cloud release

Cluster release

Support status

2.27.0

17.2.0 and 16.2.0

Deprecated for any cluster type and will be removed in one of the following releases

2.25.0

17.0.0 and 16.0.0

Generally available for managed clusters

2.24.2

15.0.1, 14.0.1, 14.0.0

Technology Preview for managed clusters

2.24.0

14.0.0

Generally available for management clusters

This section describes the MetalLBConfigTemplate custom resource used in the Container Cloud API that contains the template for MetalLB configuration for a particular cluster.

Note

The MetalLBConfigTemplate object applies to bare metal deployments only.

Before Cluster releases 17.2.0 and 16.2.0, MetalLBConfigTemplate is the default configuration method for MetalLB on bare metal deployments. This method allows the use of Subnet objects to define MetalLB IP address pools the same way as they were used before introducing the MetalLBConfig and MetalLBConfigTemplate objects. Since Cluster releases 17.2.0 and 16.2.0, use the MetalLBConfig object for this purpose instead.

For demonstration purposes, the Container Cloud MetalLBConfigTemplate custom resource description is split into the following major sections:

MetalLBConfigTemplate metadata

The Container Cloud MetalLBConfigTemplate CR contains the following fields:

  • apiVersion

    API version of the object that is ipam.mirantis.com/v1alpha1.

  • kind

    Object type that is MetalLBConfigTemplate.

The metadata object field of the MetalLBConfigTemplate resource contains the following fields:

  • name

    Name of the MetalLBConfigTemplate object.

  • namespace

    Project in which the object was created. Must match the project name of the target cluster.

  • labels

    Key-value pairs attached to the object. Mandatory labels:

    • kaas.mirantis.com/provider

      Provider type that is baremetal.

    • kaas.mirantis.com/region

      Region name that matches the region name of the target cluster.

      Note

      The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.

    • cluster.sigs.k8s.io/cluster-name

      Name of the cluster that the MetalLB configuration applies to.

    Warning

    Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.

Configuration example:

apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfigTemplate
metadata:
  name: metallb-demo
  namespace: test-ns
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: test-cluster

MetalLBConfigTemplate spec

The spec field of the MetalLBConfigTemplate object contains the templates of MetalLB configuration objects and optional auxiliary variables. Container Cloud uses these templates to create MetalLB configuration objects during the cluster deployment.

The spec field contains the following optional fields:

  • machines

    Key-value dictionary to select IpamHost objects corresponding to nodes of the target cluster. Keys contain machine aliases used in spec.templates. Values contain the NameLabelsSelector items that select IpamHost by name or by labels. For example:

    machines:
      control1:
        name: mosk-control-uefi-0
      worker1:
        labels:
          uid: kaas-node-4003a5f6-2667-40e3-aa64-ebe713a8a7ba
    

    This field is required if some IP addresses of nodes are used in spec.templates.

  • vars

    Key-value dictionary of arbitrary user-defined variables that are used in spec.templates. For example:

    vars:
      localPort: 4561
    
  • templates

    List of templates for MetalLB configuration objects that are used to render MetalLB configuration definitions and create MetalLB objects in the target cluster. Contains the following optional fields:

    • bfdProfiles

      Template for the MetalLBBFDProfile object list to create MetalLB BFDProfile objects.

    • bgpAdvertisements

      Template for the MetalLBBGPAdvertisement object list to create MetalLB BGPAdvertisement objects.

    • bgpPeers

      Template for the MetalLBBGPPeer object list to create MetalLB BGPPeer objects.

    • communities

      Template for the MetalLBCommunity object list to create MetalLB Community objects.

    • ipAddressPools

      Template for the MetalLBIPAddressPool object list to create MetalLB IPAddressPool objects.

    • l2Advertisements

      Template for the MetalLBL2Advertisement object list to create MetalLB L2Advertisement objects.

    Each template is a string and has the same structure as the list of the corresponding objects described in MetalLBConfig spec such as MetalLBIPAddressPool and MetalLBL2Advertisement, but you can use additional functions and variables inside these templates.

    Note

    When using the MetalLBConfigTemplate object, you can define MetalLB IP address pools using both Subnet objects and spec.ipAddressPools templates. IP address pools rendered from these sources will be concatenated and then written to status.renderedObjects.ipAddressPools.

    You can use the following functions in templates:

    • ipAddressPoolNames

      Selects all IP address pools of the given announcement type found for the target cluster. Possible types: layer2, bgp, any.

      The any type includes all IP address pools found for the target cluster. The announcement types of IP address pools are verified using the metallb/address-pool-protocol labels of the corresponding Subnet object.

      The ipAddressPools templates have no types as native MetalLB IPAddressPool objects have no announcement type.

      The l2Advertisements template can refer to IP address pools of the layer2 or any type.

      The bgpAdvertisements template can refer to IP address pools of the bgp or any type.

      IP address pools are searched in the templates.ipAddressPools field and in the Subnet objects of the target cluster. For example:

      l2Advertisements: |
        - name: l2services
          spec:
            ipAddressPools: {{ipAddressPoolNames "layer2"}}
      
      bgpAdvertisements: |
        - name: l3services
          spec:
            ipAddressPools: {{ipAddressPoolNames "bgp"}}
      
      l2Advertisements: |
        - name: any
          spec:
            ipAddressPools: {{ipAddressPoolNames "any"}}
      
      bgpAdvertisements: |
        - name: any
          spec:
            ipAddressPools: {{ipAddressPoolNames "any"}}
      

    The l2Advertisements object allows defining interfaces to optimize the announcement. When you use the interfaces selector, LB addresses are announced only on selected host interfaces. Mirantis recommends this configuration if nodes use separate host networks for different types of traffic. The pros of such configuration are as follows: less spam on other interfaces and networks, limited chances to reach services LB addresses from irrelevant interfaces and networks.

    Configuration example:

    l2Advertisements: |
      - name: management-lcm
        spec:
          ipAddressPools:
            - default
          interfaces:
            # LB addresses from the "default" address pool will be announced
            # on the "k8s-lcm" interface
            - k8s-lcm
    

    Caution

    Interface names in the interfaces list must match those on the corresponding nodes.

MetalLBConfigTemplate status

The status field describes the actual state of the object. It contains the following fields:

  • renderedObjects

    MetalLB objects description rendered from spec.templates in the same format as they are defined in the MetalLBConfig spec field.

    All underlying objects are optional. The following objects can be present: bfdProfiles, bgpAdvertisements, bgpPeers, communities, ipAddressPools, l2Advertisements.

  • state Since 2.23.0

    Message that reflects the current status of the resource. The list of possible values includes the following:

    • OK - object is operational.

    • ERR - object is non-operational. This status has a detailed description in the messages list.

    • TERM - object was deleted and is terminating.

  • messages Since 2.23.0

    List of error or warning messages if the object state is ERR.

  • objCreated

    Date, time, and IPAM version of the resource creation.

  • objStatusUpdated

    Date, time, and IPAM version of the last update of the status field in the resource.

  • objUpdated

    Date, time, and IPAM version of the last resource update.

MetalLB configuration examples

The following examples contain configuration templates that include MetalLBConfigTemplate.

Configuration example for using L2 (ARP) announcement

Configuration example for MetalLBConfig
apiVersion: kaas.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: kaas-mgmt
    kaas.mirantis.com/provider: baremetal
  name: mgmt-l2
  namespace: default
spec:
  templateName: mgmt-metallb-template
Configuration example for MetalLBConfigTemplate
apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfigTemplate
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: kaas-mgmt
    kaas.mirantis.com/provider: baremetal
  name: mgmt-metallb-template
  namespace: default
spec:
  templates:
    l2Advertisements: |
      - name: management-lcm
        spec:
          ipAddressPools:
            - default
          interfaces:
            # IPs from the "default" address pool will be announced on the "k8s-lcm" interface
            - k8s-lcm
      - name: provision-pxe
        spec:
          ipAddressPools:
            - services-pxe
          interfaces:
            # IPs from the "services-pxe" address pool will be announced on the "k8s-pxe" interface
            - k8s-pxe
Configuration example for Subnet of the default pool
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: kaas-mgmt
    ipam/SVC-MetalLB: ""
    kaas.mirantis.com/provider: baremetal
    metallb/address-pool-auto-assign: "true"
    metallb/address-pool-name: default
    metallb/address-pool-protocol: layer2
  name: master-lb-default
  namespace: default
spec:
  cidr: 10.0.34.0/24
  includeRanges:
  - 10.0.34.101-10.0.34.120
Configuration example for Subnet of the services-pxe pool
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: kaas-mgmt
    ipam/SVC-MetalLB: ""
    kaas.mirantis.com/provider: baremetal
    metallb/address-pool-auto-assign: "false"
    metallb/address-pool-name: services-pxe
    metallb/address-pool-protocol: layer2
  name: master-lb-pxe
  namespace: default
spec:
  cidr: 10.0.24.0/24
  includeRanges:
  - 10.0.24.221-10.0.24.230

After the objects are created and processed by the kaas-ipam Controller, the status field displays for MetalLBConfigTemplate:

Configuration example of the status field for MetalLBConfigTemplate
status:
  checksums:
    annotations: sha256:38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed
    labels: sha256:380337902278e8985e816978c349910a4f7ed98169c361eb8777411ac427e6ba
    spec: sha256:0860790fc94217598e0775ab2961a02acc4fba820ae17c737b94bb5d55390dbe
  messages:
  - Template for BFDProfiles is undefined
  - Template for BGPAdvertisements is undefined
  - Template for BGPPeers is undefined
  - Template for Communities is undefined
  objCreated: 2023-06-30T21:22:56.00000Z  by  v6.5.999-20230627-072014-ba8d918
  objStatusUpdated: 2023-07-04T00:30:35.82023Z  by  v6.5.999-20230627-072014-ba8d918
  objUpdated: 2023-06-30T22:10:51.73822Z  by  v6.5.999-20230627-072014-ba8d918
  renderedObjects:
    ipAddressPools:
    - name: default
      spec:
        addresses:
        - 10.0.34.101-10.0.34.120
        autoAssign: true
    - name: services-pxe
      spec:
        addresses:
        - 10.0.24.221-10.0.24.230
        autoAssign: false
    l2Advertisements:
    - name: management-lcm
      spec:
        interfaces:
        - k8s-lcm
        ipAddressPools:
        - default
    - name: provision-pxe
      spec:
        interfaces:
        - k8s-pxe
        ipAddressPools:
        - services-pxe
  state: OK

The following example illustrates contents of the status field that displays for MetalLBConfig after the objects are processed by the MetalLB Controller.

Configuration example of the status field for MetalLBConfig
status:
  objects:
    ipAddressPools:
    - name: default
      spec:
        addresses:
        - 10.0.34.101-10.0.34.120
        autoAssign: true
        avoidBuggyIPs: false
    - name: services-pxe
      spec:
        addresses:
        - 10.0.24.221-10.0.24.230
        autoAssign: false
        avoidBuggyIPs: false
    l2Advertisements:
    - name: management-lcm
      spec:
        interfaces:
        - k8s-lcm
        ipAddressPools:
        - default
    - name: provision-pxe
      spec:
        interfaces:
        - k8s-pxe
        ipAddressPools:
        - services-pxe
  propagateResult:
    message: Objects were successfully updated
    success: true
    time: "2023-07-05T03:10:23Z"
  updateResult:
    message: Objects were successfully read from MetalLB configuration specification
    success: true
    time: "2023-07-05T03:10:23Z"

Using the objects described above, several native MetalLB objects are created in the kaas-mgmt cluster during deployment.

Configuration example of MetalLB objects created during cluster deployment
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: management-lcm
  namespace: metallb-system
spec:
  interfaces:
  - k8s-lcm
  ipAddressPools:
  - default

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: provision-pxe
  namespace: metallb-system
spec:
  interfaces:
  - k8s-pxe
  ipAddressPools:
  - services-pxe

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default
  namespace: metallb-system
spec:
  addresses:
  - 10.0.34.101-10.0.34.120
  autoAssign: true
  avoidBuggyIPs: false

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: services-pxe
  namespace: metallb-system
spec:
  addresses:
  - 10.0.24.221-10.0.24.230
  autoAssign: false
  avoidBuggyIPs: false

Configuration example for using BGP announcement

In the following configuration example, MetalLB is configured to use BGP for announcement of external addresses of Kubernetes load-balanced services for the managed cluster from master nodes. Each master node is located in its own rack without the L2 layer extension between racks.

This section contains only examples of the objects required to illustrate the MetalLB configuration. For Rack, MultiRackCluster, L2Template and other objects required to configure BGP announcement of the cluster API load balancer address for this scenario, refer to Multiple rack configuration example.

Configuration example for MetalLBConfig
apiVersion: kaas.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: test-cluster-metallb-bgp
  namespace: managed-ns
spec:
  templateName: test-cluster-metallb-bgp-template
Configuration example for MetalLBConfigTemplate
apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfigTemplate
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: test-cluster-metallb-bgp-template
  namespace: managed-ns
spec:
  templates:
    bgpAdvertisements: |
      - name: services
        spec:
          ipAddressPools:
            - services
          peers:            # "peers" can be omitted if all defined peers
          - svc-peer-rack1  # are used in a particular "bgpAdvertisement"
          - svc-peer-rack2
          - svc-peer-rack3
    bgpPeers: |
      - name: svc-peer-rack1
        spec:
          peerAddress: 10.77.41.1  # peer address is in the external subnet #1
          peerASN: 65100
          myASN: 65101
          nodeSelectors:
            - matchLabels:
                rack-id: rack-master-1  # references the node corresponding
                                        # to the "test-cluster-master-1" Machine
      - name: svc-peer-rack2
        spec:
          peerAddress: 10.77.42.1  # peer address is in the external subnet #2
          peerASN: 65100
          myASN: 65101
          nodeSelectors:
            - matchLabels:
                rack-id: rack-master-2  # references the node corresponding
                                        # to the "test-cluster-master-2" Machine
      - name: svc-peer-rack3
        spec:
          peerAddress: 10.77.43.1  # peer address is in the external subnet #3
          peerASN: 65100
          myASN: 65101
          nodeSelectors:
            - matchLabels:
                rack-id: rack-master-3  # references the node corresponding
                                        # to the "test-cluster-master-3" Machine
Configuration example for Subnet
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    ipam/SVC-MetalLB: ""
    kaas.mirantis.com/provider: baremetal
    metallb/address-pool-auto-assign: "true"
    metallb/address-pool-name: services
    metallb/address-pool-protocol: bgp
  name: test-cluster-lb
  namespace: managed-ns
spec:
  cidr: 134.33.24.0/24
  includeRanges:
    - 134.33.24.221-134.33.24.240

The following objects illustrate configuration for three subnets that are used to configure external network in three racks. Each master node uses its own external L2/L3 network segment.

Configuration example for the Subnet ext-rack-control-1
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: ext-rack-control-1
  namespace: managed-ns
spec:
  cidr: 10.77.41.0/28
  gateway: 10.77.41.1
  includeRanges:
    - 10.77.41.3-10.77.41.13
  nameservers:
    - 1.2.3.4
Configuration example for the Subnet ext-rack-control-2
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: ext-rack-control-2
  namespace: managed-ns
spec:
  cidr: 10.77.42.0/28
  gateway: 10.77.42.1
  includeRanges:
    - 10.77.42.3-10.77.42.13
  nameservers:
    - 1.2.3.4
Configuration example for the Subnet ext-rack-control-3
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
  name: ext-rack-control-3
  namespace: managed-ns
spec:
  cidr: 10.77.43.0/28
  gateway: 10.77.43.1
  includeRanges:
    - 10.77.43.3-10.77.43.13
  nameservers:
    - 1.2.3.4

Rack objects and ipam/RackRef labels in Machine objects are not required for MetalLB configuration. But in this example, rack objects are implied to be used for configuration of BGP announcement of the cluster API load balancer address. Rack objects are not present in this example.

Machine objects select different L2 templates because each master node uses different L2/L3 network segments for LCM, external, and other networks.

Configuration example for the Machine test-cluster-master-1
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-1
  namespace: managed-ns
  annotations:
    metal3.io/BareMetalHost: managed-ns/test-cluster-master-1
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master-1
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-1
      l2TemplateSelector:
        name: test-cluster-master-1
      nodeLabels:
      - key: rack-id          # it is used in "nodeSelectors"
        value: rack-master-1  # of "bgpPeer" MetalLB objects
Configuration example for the Machine test-cluster-master-2
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-2
  namespace: managed-ns
  annotations:
    metal3.io/BareMetalHost: managed-ns/test-cluster-master-2
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master-2
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-2
      l2TemplateSelector:
        name: test-cluster-master-2
      nodeLabels:
      - key: rack-id          # it is used in "nodeSelectors"
        value: rack-master-1  # of "bgpPeer" MetalLB objects
Configuration example for the Machine test-cluster-master-2
apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-master-3
  namespace: managed-ns
  annotations:
    metal3.io/BareMetalHost: managed-ns/test-cluster-master-3
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    cluster.sigs.k8s.io/control-plane: controlplane
    hostlabel.bm.kaas.mirantis.com/controlplane: controlplane
    ipam/RackRef: rack-master-3
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      kind: BareMetalMachineProviderSpec
      apiVersion: baremetal.k8s.io/v1alpha1
      hostSelector:
        matchLabels:
          kaas.mirantis.com/baremetalhost-id: test-cluster-master-3
      l2TemplateSelector:
        name: test-cluster-master-3
      nodeLabels:
      - key: rack-id          # it is used in "nodeSelectors"
        value: rack-master-3  # of "bgpPeer" MetalLB objects