Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Now, the MOSK documentation set covers all product layers, including MOSK management (formerly Container Cloud). This means everything you need is in one place. Some legacy names may remain in the code and documentation and will be updated in future releases. The separate Container Cloud documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Separate PXE and management (LCM) networks¶

This section describes how to configure a dedicated PXE network for a management bare metal cluster. A separate PXE network allows isolating sensitive bare metal provisioning process from the end users. The users still have access to MOSK management services, such as Keycloak, to authenticate workloads in MOSK clusters, for example, using Horizon.

Note

This additional configuration procedure must be completed as part of the main Deploy a management cluster using CLI procedure. It substitutes or appends some configuration parameters and templates that are used in the main procedure for the management cluster to use two networks, PXE and management (LCM), instead of combined networks. Mirantis recommends considering the main procedure first.

The following table describes the overall network mapping scheme with all L2/L3 parameters, for example, for two networks, PXE (CIDR 10.0.0.0/24) and management (CIDR 10.0.11.0/24):

Network mapping overview¶
Deployment file name	Network	Parameters and values
`ipam-objects.yaml`	PXE	`SET_PXE_CIDR=10.0.0.0/24` `SET_PXE_ADDR_POOL=10.0.0.100-10.0.0.109` `SET_PXE_ADDR_RANGE=10.0.0.110-10.0.0.120`
`ipam-objects.yaml`	Management	`SET_LCM_CIDR=10.0.11.0/24` `SET_LCM_RANGE=10.0.11.100-10.0.11.199` `SET_LCM_GW=10.0.11.1` `SET_LCM_DNS=8.8.8.8` `SET_LB_HOST=10.0.11.90`
`metallbconfig.yaml`	MetalLB	`SET_METALLB_PXE_ADDR_POOL=10.0.0.61-10.0.0.70` `SET_METALLB_ADDR_POOL=10.0.11.61-10.0.11.80`
`bootstrap.sh`	PXE	`KAAS_BM_PXE_IP=10.0.0.20` `KAAS_BM_PXE_MASK=24` `KAAS_BM_PXE_BRIDGE=br0`

When using separate PXE and management networks, the management cluster services are exposed in different networks using two separate MetalLB address pools:

Services exposed through the PXE network are as follows:
- Ironic API as a bare metal provisioning server
- HTTP server that provides images for network boot and server provisioning
- Caching server for accessing the MOSK management artifacts deployed on hosts
Services exposed through the management network are all other MOSK management services, such as Keycloak, management console, and so on.

To configure separate PXE and management LCM networks:

To ensure successful bootstrap, enable asymmetric routing on the interfaces of the management cluster nodes. This is required because the seed node relies on one network by default, which can potentially cause traffic asymmetry.

In the kernelParameters section of baremetalhostprofiles.yaml.template, set rp_filter to 2. This enables loose mode as defined in RFC3704.
Example configuration of asymmetric routing
... kernelParameters: ... sysctl: # Enables the "Loose mode" for the "k8s-lcm" interface (management network) net.ipv4.conf.k8s-lcm.rp_filter: "2" # Enables the "Loose mode" for the "bond0" interface (PXE network) net.ipv4.conf.bond0.rp_filter: "2" ...
Note

More complicated solutions that are not described in this manual include getting rid of traffic asymmetry, for example:
- Configure source routing on management cluster nodes.
- Plug the seed node into the same networks as the management cluster nodes, which requires custom configuration of the seed node.

In kaas-bootstrap/templates/bm/ipam-objects.yaml.template:

Substitute all Subnet object templates with the new ones as described in the example template below
Update the L2 template spec.l3Layout and spec.npTemplate fields as described in the example template below

Example of the Subnet object templates

# Subnet object that provides IP addresses using DHCP for bare metal
# hosts of the management cluster in the PXE network during provisioning.
apiVersion: "ipam.mirantis.com/v1alpha1"
kind: Subnet
metadata:
  name: mgmt-pxe
  namespace: default
  labels:
    kaas.mirantis.com/provider: baremetal
    ipam/SVC-dhcp-range: "presents"
spec:
  cidr: SET_PXE_CIDR
  includeRanges:
    - SET_PXE_ADDR_POOL
  excludeRanges:
    - SET_PXE_ADDR_RANGE
    - SET_METALLB_PXE_ADDR_POOL
---
# Subnet object that provides IP addresses for bare metal hosts of the
# management cluster in the PXE network after provisioning.
apiVersion: "ipam.mirantis.com/v1alpha1"
kind: Subnet
metadata:
  name: mgmt-pxe-nics
  namespace: default
  labels:
    kaas.mirantis.com/provider: baremetal
    ipam/SVC-pxe-nics: "presents"
    cluster.sigs.k8s.io/cluster-name: CLUSTER_NAME
spec:
  cidr: SET_PXE_CIDR
  includeRanges:
    - SET_PXE_ADDR_RANGE
  excludeRanges:
    - SET_PXE_ADDR_POOL
    - SET_METALLB_PXE_ADDR_POOL
---
# Subnet object that provides IP addresses for bare metal hosts of the
# management cluster in the management network.
apiVersion: "ipam.mirantis.com/v1alpha1"
kind: Subnet
metadata:
  name: mgmt-lcm
  namespace: default
  labels:
    kaas.mirantis.com/provider: baremetal
    ipam/SVC-k8s-lcm: "presents"
    cluster.sigs.k8s.io/cluster-name: CLUSTER_NAME
spec:
  cidr: {{ SET_LCM_CIDR }}
  gateway: SET_LCM_GW
  nameservers:
  - SET_LCM_DNS
  includeRanges:
    - {{ SET_LCM_RANGE }}
  excludeRanges:
    - SET_LB_HOST
    - SET_METALLB_ADDR_POOL
---
# MetalLBConfig object that provides configuration
# for "services-pxe" and "default" MetalLB address pools that are
# used to expose LB endpoints of services in the PXE and management
# networks accordingly.
apiVersion: kaas.mirantis.com/v1alpha1
kind: MetalLBConfig
metadata:
  name: kaas-mgmt-metallb
  namespace: default
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: CLUSTER_NAME
spec:
  ipAddressPools:
  - name: default
    spec:
      addresses:
      - SET_METALLB_ADDR_POOL
      autoAssign: true
      avoidBuggyIPs: false
  - name: services-pxe
    spec:
      addresses:
      # IP address range to use for the MetalLB address pool to expose
      # Kubernetes services in the PXE network.
      # This address range must be within the PXE network.
      # The minimum required address range size is 5 IP addresses.
      - SET_METALLB_PXE_ADDR_POOL
      autoAssign: false
      avoidBuggyIPs: false
  l2Advertisements:
  - name: default
    spec:
      interfaces:
      - k8s-lcm
      ipAddressPools:
      - default
  - name: pxe
    spec:
      interfaces:
      - bm-pxe
      ipAddressPools:
      - services-pxe

Warning

The bm-pxe address must have a separate interface with only one address on this interface.

Verify the current MetalLB configuration that is stored in MetalLB objects:
```
kubectl -n metallb-system get ipaddresspools,l2advertisements
```
For the example configuration described above, the system outputs similar content:
```
NAME                                    AGE
ipaddresspool.metallb.io/default        129m
ipaddresspool.metallb.io/services-pxe   129m

NAME                                      AGE
l2advertisement.metallb.io/default        129m
l2advertisement.metallb.io/services-pxe   129m
```
To verify the MetalLB objects:
```
kubectl -n metallb-system get <object> -o json | jq '.spec'
```
For the example configuration described above, the system outputs a similar content for ipaddresspool objects:
```
{
  "addresses": [
    "10.0.11.61-10.0.11.80"
  ],
  "autoAssign": true,
  "avoidBuggyIPs": false
}
$ kubectl -n metallb-system get ipaddresspool.metallb.io/services-pxe -o json | jq '.spec'
{
  "addresses": [
    "10.0.0.61-10.0.0.70"
  ],
  "autoAssign": false,
  "avoidBuggyIPs": false
}
```
The auto-assign parameter will be set to false for all address pools except the default one. So, a particular service will get an address from such an address pool only if the Service object has a special metallb.universe.tf/address-pool annotation that points to the specific address pool name.
Note

It is expected that every MOSK management service on a management cluster will be assigned to one of the address pools. Current consideration is to have two MetalLB address pools:
- services-pxe is a reserved address pool name to use for the MOSK management services in the PXE network (Ironic API, HTTP server, caching server).
  
  The bootstrap cluster also uses the services-pxe address pool for its provision services for management cluster nodes to be provisioned from the bootstrap cluster. After the management cluster is deployed, the bootstrap cluster is deleted and that address pool is solely used by the newly deployed cluster.
- default is an address pool to use for all other MOSK management services in the management network. No annotation is required on the Service objects in this case.

In addition to the network parameters defined in Deploy a management cluster using CLI, configure the following MetalLB parameters by replacing them in templates/bm/metallbconfig.yaml.template:

MetalLB parameters¶
Parameter	Description	Example value
`SET_METALLB_PXE_ADDR_POOL`	Address range to be used for LB endpoints of the MOSK management services in the PXE network: Ironic API, HTTP server, and caching server. This range must be within the PXE network. The minimum required range is 5 IP addresses.	`10.0.0.61-10.0.0.70`
`SET_METALLB_ADDR_POOL`	The address range to be used for the externally accessible LB endpoints of the MOSK management services, such as Keycloak, management console, and so on. This range must be within the management network. The minimum required range is 19 IP addresses. This range must NOT include the `SET_LB_HOST` address.	`10.0.11.61-10.0.11.80`

Proceed to further steps in Deploy a management cluster using CLI.