Separate PXE and management networks¶

This section describes how to configure a dedicated PXE network for a management or regional bare metal cluster. A separate PXE network allows isolating sensitive bare metal provisioning process from the end users. The users still have access to Container Cloud services, such as Keycloak, to authenticate workloads in managed clusters, such as Horizon in a Mirantis OpenStack for Kubernetes cluster.

Note

This additional configuration procedure must be completed as part of the Prepare metadata and deploy the management cluster steps. It substitutes or appends some configuration parameters and templates that are used in Prepare metadata and deploy the management cluster for the management cluster to use two networks, PXE and management, instead of one PXE/management network. We recommend considering the Prepare metadata and deploy the management cluster procedure first.

The following table describes the overall network mapping scheme with all L2/L3 parameters, for example, for two networks, PXE (CIDR 10.0.0.0/24) and management (CIDR 10.0.11.0/24):

Network mapping overview¶
Deployment file name	Network	Parameters and values
`cluster.yaml`	Management	`SET_LB_HOST=10.0.11.90` `SET_METALLB_ADDR_POOL=10.0.11.61-10.0.11.80`
`ipam-objects.yaml`	PXE	`SET_IPAM_CIDR=10.0.0.0/24` `SET_PXE_NW_GW=10.0.0.1` `SET_PXE_NW_DNS=8.8.8.8` `SET_IPAM_POOL_RANGE=10.0.0.100-10.0.0.109` `SET_METALLB_PXE_ADDR_POOL=10.0.0.61-10.0.0.70`
`ipam-objects.yaml`	Management	`SET_LCM_CIDR=10.0.11.0/24` `SET_LCM_RANGE=10.0.11.100-10.0.11.199` `SET_LB_HOST=10.0.11.90` `SET_METALLB_ADDR_POOL=10.0.11.61-10.0.11.80`
`bootstrap.sh`	PXE	`KAAS_BM_PXE_IP=10.0.0.20` `KAAS_BM_PXE_MASK=24` `KAAS_BM_PXE_BRIDGE=br0` `KAAS_BM_BM_DHCP_RANGE=10.0.0.30,10.0.0.59,255.255.255.0` `BOOTSTRAP_METALLB_ADDRESS_POOL=10.0.0.61-10.0.0.80`

When using a separate PXE network, the management cluster services are exposed in different networks using two separate MetalLB address pools:

Services exposed through the PXE network are as follows:
- Ironic API (bare metal provisioning server)
- HTTP server that provides images for network boot and server provisioning
- Caching server for accessing the Container Cloud artifacts deployed on hosts
Services exposed through the management network are all other Container Cloud services, such as Keycloak, web UI, and so on

To configure separate PXE and management networks:

In kaas-bootstrap/templates/bm/ipam-objects.yaml.template:

Substitute all the Subnet object templates with the new ones as described in the example template below
Update the L2 template spec.l3Layout and spec.npTemplate fields as described in the example template below

The last Subnet template named mgmt-pxe-lb in the example above will be used to configure the MetalLB address pool in the PXE network. The bare metal provider will automatically configure MetalLB with address pools using the Subnet objects identified by specific labels.

Warning

The bm-pxe address must have a separate interface with only one address on this interface.

Use the following labels to identify the Subnet object as a MetalLB address pool and configure the name and protocol for that address pool. All labels below are mandatory for the Subnet object that configures a MetalLB address pool.

Mandatory Subnet labels for a MetalLB address pool¶
Label	Description
`ipam/SVC-MetalLB`	Defines that the `Subnet` object will be used to provide a new address pool/range for MetalLB.
`metallb/address-pool-name`	Sets the name `services-pxe` for the newly created address pool. The `services-pxe` address pool name is mandatory when configuring a dedicated PXE network in the management cluster. This name will be used in annotations for services exposed through the PXE network. Every address pool must have a distinct name except the `default` name that is reserved for the management network.
`metallb/address-pool-auto-assign`	Configures the auto-assign policy of an address pool. Boolean. Is set to `true` and is not configurable for address pools defined through the cluster specification. For any service that does not have a specific MetalLB annotation configured, MetalLB allocates external IPs from arbitrary address pools that have the auto-assign policy set to `true`. Only for the service that has a specific MetalLB annotation with the address pool name, MetalLB allocates external IPs from the address pool having the auto-assign policy set to `false`.
`metallb/address-pool-protocol`	Sets the address pool protocol. The only supported value is `layer2` (default).
`cluster.sigs.k8s.io/cluster-name`	Specifies the management or regional cluster name that the `Subnet` should be bound to.

Caution

Do not set the same address pool name for two or more Subnet objects. Otherwise, the corresponding MetalLB address pool configuration fails with a warning message in the bare metal provider log.

Caution

At least one MetalLB address pool must have the auto-assign policy enabled so that unannotated services can have load balancer IPs allocated for them. To satisfy this requirement, either configure one of address pools using the cluster specification or configure it using Subnet with metallb/address-pool-auto-assign: "true".
When configuring multiple address pools with the auto-assign policy enabled, keep in mind that it is not determined which of those address pools will be used to allocate an IP for a particular unannotated service.

Verify the current MetalLB configuration:
Since Container Cloud 2.21.0
The MetalLB configuration is stored in MetalLB objects:
kubectl -n metallb-system get ipaddresspools,l2advertisements
For the example configuration described above, the system outputs a similar content:
NAME AGE ipaddresspool.metallb.io/default 129m ipaddresspool.metallb.io/services-pxe 129m NAME AGE l2advertisement.metallb.io/default 129m l2advertisement.metallb.io/services-pxe 129m
Verify the MetalLB objects:
kubectl -n metallb-system get <object> -o json | jq '.spec'
For the example configuration described above, the system outputs a similar content for ipaddresspool objects:
{ "addresses": [ "10.0.11.61-10.0.11.80" ], "autoAssign": true, "avoidBuggyIPs": false } $ kubectl -n metallb-system get ipaddresspool.metallb.io/services-pxe -o json | jq '.spec' { "addresses": [ "10.0.0.61-10.0.0.70" ], "autoAssign": false, "avoidBuggyIPs": false }
Before Container Cloud 2.21.0
The MetalLB configuration is stored in the ConfigMap object:
kubectl -n metallb-system get cm metallb -o jsonpath={.data.config}
For the example configuration described above, a successful output is as follows:
address-pools: - name: default protocol: layer2 addresses: - 10.0.11.61-10.0.11.80 - name: services-pxe protocol: layer2 auto-assign: false addresses: - 10.0.0.61-10.0.0.70
The auto-assign parameter will be set to false for all address pools except the default one. So, a particular service will get an address from such an address pool only if the Service object has a special metallb.universe.tf/address-pool annotation that points to the specific address pool name.
Note

It is expected that every Container Cloud service on a management and regional cluster will be assigned to one of the address pools. Current consideration is to have two MetalLB address pools:
- services-pxe is a reserved address pool name to use for the Container Cloud services in the PXE network (Ironic API, HTTP server, caching server)
- default is an address pool to use for all other Container Cloud services in the management network. No annotation is required on the Service objects in this case.
In kaas-bootstrap/templates/bm/cluster.yaml.template, add the dedicatedMetallbPools flag and set it to true:
```
spec:
  ...
  providerSpec:
    value:
      apiVersion: baremetal.k8s.io/v1alpha1
      kind: BaremetalClusterProviderSpec
      ...
      dedicatedMetallbPools: true
      ...
```
User sets this flag to enable splitting of LB endpoints for the Container Cloud services. The metallb.universe.tf/address-pool annotations on the Service objects are configured by the bare metal provider automatically when the dedicatedMetallbPools flag is set to true.

Example Service object configured by the baremetal-operator Helm release:
```
apiVersion: v1
kind: Service
metadata:
  name: ironic-api
  annotations:
    metallb.universe.tf/address-pool: services-pxe
spec:
  ports:
  - port: 443
    targetPort: 443
  type: LoadBalancer
```
The metallb.universe.tf/address-pool annotation on the Service object is set to services-pxe by the baremetal provider, so the ironic-api service will be assigned an LB address from the corresponding MetalLB address pool.

In addition to the network parameters defined in Prepare metadata and deploy the management cluster, configure the following ones by replacing them in templates/bm/ipam-objects.yaml.template:

New subnet template parameters¶
Parameter	Description	Example value
`SET_LCM_CIDR`	Address of a management network for the management cluster in the CIDR notation. You can later share this network with managed clusters where it will act as the LCM network. If managed clusters have their separate LCM networks, those networks must be routable to the management network.	`10.0.11.0/24`
`SET_LCM_RANGE`	Address range that includes addresses to be allocated to bare metal hosts in the management network for the management cluster. When this network is shared with managed clusters, the size of this range limits the number of hosts that can be deployed in all clusters that share this network. When this network is solely used by a management cluster, the range should include at least 3 IP addresses for bare metal hosts of the management cluster.	`10.0.11.100-10.0.11.109`
`SET_METALLB_PXE_ADDR_POOL`	Address range to be used for LB endpoints of the Container Cloud services: Ironic-API, HTTP server, and caching server. This range must be within the PXE network. The minimum required range is 5 IP addresses.	`10.0.0.61-10.0.0.70`

The following parameters will now be tied to the management network while their meaning remains the same as described in Prepare metadata and deploy the management cluster:

Subnet template parameters migrated to management network¶
Parameter	Description	Example value
`SET_LB_HOST`	IP address of the externally accessible API endpoint of the management cluster. This address must NOT be within the `SET_METALLB_ADDR_POOL` range but within the management network. External load balancers are not supported.	`10.0.11.90`
`SET_METALLB_ADDR_POOL`	The address range to be used for the externally accessible LB endpoints of the Container Cloud services, such as Keycloak, web UI, and so on. This range must be within the management network. The minimum required range is 19 IP addresses.	`10.0.11.61-10.0.11.80`

Proceed to further steps in Prepare metadata and deploy the management cluster.