Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Now, the MOSK documentation set covers all product layers, including MOSK management (formerly Container Cloud). This means everything you need is in one place. Some legacy names may remain in the code and documentation and will be updated in future releases. The separate Container Cloud documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Configure and verify MetalLB

This section describes how to set up and verify MetalLB parameters before configuring subnets for a MOSK cluster.

Caution

This section also applies to the bootstrap procedure of a management cluster with the following differences:

  • Instead of the Cluster object, configure templates/bm/cluster.yaml.template.

  • Instead of the MetalLBConfig object, configure templates/bm/metallbconfig.yaml.template.

  • Instead of creating specific IPAM objects such as Subnet and L2Template (as well as Rack and MultiRackCluster when using BGP configuration), add their settings to templates/bm/ipam-objects.yaml.template.

Configuration rules for the ‘MetalLBConfig’ object

Caution

The use of the MetalLBConfig object is mandatory.

The following rules and requirements apply to configuration of the MetalLBConfig object:

  • Define one MetalLBConfig object per cluster.

  • Define the following mandatory labels:

    cluster.sigs.k8s.io/cluster-name

    Specifies the cluster name where the MetalLB address pool is used.

    kaas.mirantis.com/provider

    Specifies the provider of the cluster where the MetalLB address pool is used.

  • Intersection of IP address ranges within any single MetalLB address pool is not allowed.

  • At least one MetalLB address pool must have the auto-assign policy enabled so that unannotated services can have load balancer IP addresses allocated for them.

  • When configuring multiple address pools with the auto-assign policy enabled, keep in mind that it is not determined in advance which pool of those multiple address pools is used to allocate an IP address for a particular unannotated service.

Note

You can optimize address announcement for load-balanced services using the interfaces selector for the l2Advertisements object. This selector allows for address announcement only on selected host interfaces. For details, see MetalLBConfig spec.

Configure and verify MetalLB using MOSK management console

Note

The BGP configuration is not yet supported in the MOSK management console. Meantime, use the CLI for this purpose. For details, see Configure and verify MetalLB using the CLI.

  1. Read the MetalLB configuration guidelines described in Configuration rules for the ‘MetalLBConfig’ object.

  2. Optional. Configure parameters related to MetalLB components life cycle such as deployment and update using the metallb Helm chart values in the Cluster spec section. For example:

  3. Log in to the MOSK management console with the writer permissions.

  4. Switch to the required non-default project using the Switch Project action icon located on top of the main left-side navigation panel.

    Caution

    Do not create a MOSK cluster in the default project (Kubernetes namespace), which is dedicated for the management cluster only. If no projects are defined, first create a new mosk project as described in Create a project for MOSK clusters.

  5. In the Networks section, click the MetalLB Configs tab.

  6. Click Create MetalLB Config.

  7. Fill out the Create MetalLB Config form as required:

    • Name

      Name of the MetalLB object being created.

    • Cluster

      Name of the cluster that the MetalLB object is being created for

    • IP Address Pools

      List of MetalLB IP address pool descriptions that will be used to create the MetalLB IPAddressPool objects. Click the + button on the right side of the section to add more objects.

      • Name

        IP address pool name.

      • Addresses

        Comma-separated ranges of the IP addresses included into the address pool.

      • Auto Assign

        Enable auto-assign policy for unannotated services to have load balancer IP addresses allocated to them. At least one MetalLB address pool must have the auto-assign policy enabled.

      • Service Allocation

        IP address pool allocation to services. Click Edit to insert a service allocation object with required label selectors for services in the YAML format. For example:

        serviceSelectors:
- matchExpressions:
  - key: app.kubernetes.io/name
    operator: NotIn
    values:
    - dhcp-lb

        For details on the MetalLB IPAddressPool object type, see MetalLB documentation.

      • L2 Advertisements

        List of MetalLBL2Advertisement objects to create MetalLB L2Advertisement objects.

        The l2Advertisements object allows defining interfaces to optimize the announcement. When you use the interfaces selector, LB addresses are announced only on selected host interfaces.

        Mirantis recommends using the interfaces selector if nodes use separate host networks for different types of traffic. The pros of such configuration are as follows: less spam on other interfaces and networks and limited chances to reach IP addresses of load-balanced services from irrelevant interfaces and networks.

        Caution

        Interface names in the interfaces list must match those on the corresponding nodes.

        Add the following parameters:

        • Name

          Name of the l2Advertisements object.

        • Interfaces

          Optional. Comma-separated list of interface names that must match the ones on the corresponding nodes. These names are defined in L2 templates that are linked to the selected cluster.

        • IP Address Pools

          Select the IP adress pool to use for the l2Advertisements object.

        • Node Selectors

          Optional. Match labels and values for the Kubernetes node selector to limit the nodes announced as next hops for the LoadBalancer IP. If you do not provide any labels, all nodes are announced as next hops.

        For details on the MetalLB L2Advertisements object type, see MetalLB documentation.

  8. Click Create.

  9. In Networks > MetalLB Configs, verify the status of the created MetalLB object:

    • Ready - object is operational.

    • Error - object is non-operational. Hover over the status to obtain details of the issue.

    Note

    To verify the object details, in Networks > MetalLB Configs, click the More action icon in the last column of the required object section and select MetalLB Config info.

  10. Proceed to creating cluster subnets as described in Create subnets.

Configure and verify MetalLB using the CLI

  1. Optional. Configure parameters related to MetalLB components life cycle such as deployment and update using the metallb Helm chart values in the Cluster spec section. For example:

  2. Configure the MetalLB parameters related to IP address allocation and announcement for load-balanced cluster services:

    1. Create the MetalLBConfig object:

    2. In the Technology Preview scope, you can use BGP for announcement of external addresses of Kubernetes load-balanced services for a MOSK cluster. To configure the BGP announcement mode for MetalLB, use MetalLBConfig object.

      The use of BGP is required to announce IP addresses for load-balanced services when using MetalLB on nodes that are distributed across multiple racks. In this case, setting of rack-id labels on nodes is required, they are used in node selectors for BGPPeer, BGPAdvertisement, or both MetalLB objects to properly configure BGP connections from each node.

      Configuration example of the Machine object for the BGP announcement mode
      apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
  name: test-cluster-compute-1
  namespace: mosk-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    ipam/RackRef: rack-1  # reference to the "rack-1" Rack
    kaas.mirantis.com/provider: baremetal
spec:
  providerSpec:
    value:
      ...
      nodeLabels:
      - key: rack-id   # node label can be used in "nodeSelectors" inside
        value: rack-1  # "BGPPeer" and/or "BGPAdvertisement" MetalLB objects
  ...
      Configuration example of the MetalLBConfig object for the BGP announcement mode
      apiVersion: ipam.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
  name: test-cluster-metallb-config
  namespace: mosk-ns
  labels:
    cluster.sigs.k8s.io/cluster-name: test-cluster
    kaas.mirantis.com/provider: baremetal
spec:
  ...
  bgpPeers:
    - name: svc-peer-1
      spec:
        holdTime: 0s
        keepaliveTime: 0s
        peerAddress: 10.77.42.1
        peerASN: 65100
        myASN: 65101
        nodeSelectors:
          - matchLabels:
            rack-id: rack-1  # references the nodes having
                             # the "rack-id=rack-1" label
  bgpAdvertisements:
    - name: services
      spec:
        aggregationLength: 32
        aggregationLengthV6: 128
        ipAddressPools:
          - services
        peers:
          - svc-peer-1
          ...

      The bgpPeers and bgpAdvertisements fields are used to configure BGP announcement instead of l2Advertisements.

      The use of BGP for announcement also allows for better balancing of service traffic between cluster nodes as well as gives more configuration control and flexibility for infrastructure administrators. For configuration examples, refer to MetalLB configuration examples. For configuration procedure, refer to Configure BGP announcement for cluster API LB address.

  3. Verify the current MetalLB configuration that is stored in MetalLB objects:

    kubectl -n metallb-system get ipaddresspools,l2advertisements

    The example system output:

    NAME                                    AGE
ipaddresspool.metallb.io/default        129m
ipaddresspool.metallb.io/services-pxe   129m

NAME                                      AGE
l2advertisement.metallb.io/default        129m
l2advertisement.metallb.io/services-pxe   129m

    Verify one of the listed above MetalLB objects:

    kubectl -n metallb-system get <object> -o json | jq '.spec'

    The example system output for ipaddresspool objects:

    $ kubectl -n metallb-system get ipaddresspool.metallb.io/default -o json | jq '.spec'
{
  "addresses": [
    "10.0.11.61-10.0.11.80"
  ],
  "autoAssign": true,
  "avoidBuggyIPs": false
}
$ kubectl -n metallb-system get ipaddresspool.metallb.io/services-pxe -o json | jq '.spec'
{
  "addresses": [
    "10.0.0.61-10.0.0.70"
  ],
  "autoAssign": false,
  "avoidBuggyIPs": false
}

    The autoAssign parameter will be set to false for all address pools except the default one. So, a particular service will get an address from such an address pool only if the Service object has a special metallb.universe.tf/address-pool annotation that points to the specific address pool name.

    Note

    It is expected that every Kubernetes service on a management cluster will be assigned to one of the address pools. Current consideration is to have two MetalLB address pools:

    • services-pxe is a reserved address pool name to use for the Kubernetes services in the PXE network (Ironic API, HTTP server, caching server).

    • default is an address pool to use for all other Kubernetes services in the management network. No annotation is required on the Service objects in this case.

  4. Proceed to creating cluster subnets as described in Create subnets.