Create subnets for a multi-rack MOSK cluster

When planning your installation in advance, you need to prepare a set of subnets and L2 templates for every rack in your cluster. For details, see Multi-rack architecture.

Prerequisites

Create subnets for a multi-rack cluster

Note

In this section, the exemplary cluster name is mosk-cluster-name. Adjust it to fit your deployment.

  1. Log in to a local machine where your management cluster kubeconfig is located and where kubectl is installed.

    Note

    The management cluster kubeconfig is created during the last stage of the management cluster bootstrap.

  2. Create the subnet.yaml file with a number of subnets depending on the configuration of your MOSK cluster:

    kubectl --kubeconfig <pathToManagementClusterKubeconfig> apply -f <SubnetFileName.yaml>

  3. Using the Subnet object examples for a multi-rack cluster that are described in the following sections, create subnets for the target cluster.

Note

Subnet labels such as rack-x-lcm, rack-api-lcm, and so on are optional. You can use them in L2 templates to select Subnet objects by label.

Note

Before the Cluster release 16.1.0, the Subnet object contains the kaas.mirantis.com/region label that specifies the region where the Subnet object will be applied.

Provisioning network

Configure DHCP relay agents on the edges of the broadcast domains in the provisioning network, as needed.

Make sure to assign the IP address ranges you want to allocate to the hosts using DHCP for discovery and inspection. Create subnets using these IP parameters. Specify the IP address of your DHCP relay as the default gateway in the corresponding Subnet object.

Caution

Support of multiple DHCP ranges has the following limitations:

  • Using of custom DNS server addresses for servers that boot over PXE is not supported.

  • The Subnet objects for DHCP ranges cannot be associated with any specific cluster, as the DHCP server configuration is only applicable to the management cluster where the DHCP server is running. The cluster.sigs.k8s.io/cluster-name label will be ignored.

Example mos-racks-dhcp-subnets.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-dhcp
  namespace: default
  labels:
    ipam/SVC-dhcp-range: "1"
    kaas.mirantis.com/provider: baremetal
spec:
  cidr: 10.20.101.0/24
  gateway: 10.20.101.1
  includeRanges:
    - 10.20.101.16-10.20.101.127
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-dhcp
  namespace: default
  labels:
    ipam/SVC-dhcp-range: "1"
    kaas.mirantis.com/provider: baremetal
spec:
  cidr: 10.20.102.0/24
  gateway: 10.20.102.1
  includeRanges:
    - 10.20.102.16-10.20.102.127
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-dhcp
  namespace: default
  labels:
    ipam/SVC-dhcp-range: "1"
    kaas.mirantis.com/provider: baremetal
spec:
  cidr: 10.20.103.0/24
  gateway: 10.20.103.1
  includeRanges:
    - 10.20.103.16-10.20.103.127
---
# Add more Subnet object templates as required using the above example
# (one subnet per rack)

LCM network

This is the IP address space that Container Cloud uses to ensure communication between the LCM agents and the management API. These addresses are also used by Kubernetes nodes for communication. The addresses from the subnets are assigned to all MOSK cluster nodes.

Example mosk-racks-lcm-subnets.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-lcm
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-k8s-lcm: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-lcm: "true"
spec:
  cidr: 10.20.111.0/24
  gateway: 10.20.111.1
  includeRanges:
    - 10.20.111.16-10.20.111.255
  nameservers:
    - 8.8.8.8
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-lcm
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-k8s-lcm: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-lcm: "true"
spec:
  cidr: 10.20.112.0/24
  gateway: 10.20.112.1
  includeRanges:
    - 10.20.112.16-10.20.112.255
  nameservers:
    - 8.8.8.8
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-lcm
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-k8s-lcm: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-lcm: "true"
spec:
  cidr: 10.20.113.0/24
  gateway: 10.20.113.1
  includeRanges:
    - 10.20.113.16-10.20.113.255
  nameservers:
    - 8.8.8.8
---
# Add more subnet object templates as required using the above example
# (one subnet per rack)

API/LCM network

Note

Since 23.2.2, MOSK supports full L3 networking topology in the Technology Preview scope. This enables deployment of specific cluster segments in dedicated racks without the need for L2 layer extension between them. For configuration procedure, see Configure BGP announcement for cluster API LB address and Configure BGP announcement of external addresses of Kubernetes load-balanced services in Deployment Guide.

If BGP announcement is configured for the MOSK cluster API LB address, the API/LCM network is not required. Announcement of the cluster API LB address is done using the LCM network.

If you configure ARP announcement of the load-balancer IP address for the MOSK cluster API, the API/LCM network must be configured on the Kubernetes manager nodes of the cluster. This network contains the Kubernetes API endpoint with the VRRP virtual IP address.

This network contains Kubernetes API endpoint with the VRRP virtual IP address. This is the IP address space that Container Cloud uses to ensure communication between the LCM agents and the management API. These addresses are also used by Kubernetes nodes for communication. The addresses from the subnet are assigned to all Kubernetes manager nodes of the MOSK cluster.

Example mosk-racks-api-lcm-subnet.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-api-lcm
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-k8s-lcm: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-api-lcm: "true"
spec:
  cidr: 10.20.110.0/24
  gateway: 10.20.110.1
  includeRanges:
    - 10.20.110.16-10.20.110.25
  nameservers:
    - 8.8.8.8

External network

The network is used to expose the OpenStack, StackLight, and other services of the MOSK cluster.

Note

Since 23.2.2, MOSK supports full L3 networking topology in the Technology Preview scope. This enables deployment of specific cluster segments in dedicated racks without the need for L2 layer extension between them. For configuration procedure, see Configure BGP announcement for cluster API LB address and Configure BGP announcement of external addresses of Kubernetes load-balanced services in Deployment Guide.

If you configure BGP announcement for IP addresses of load-balanced services of a MOSK cluster, the external network can consist of multiple VLAN segments connected to all nodes of a MOSK cluster where MetalLB speaker components are configured to announce IP addresses for Kubernetes load-balanced services. Mirantis recommends that you use OpenStack controller nodes for this purpose.

If you configure ARP announcement for IP addresses of load-balanced services of a MOSK cluster, the external network must consist of a single VLAN stretched to the ToR switches of all the racks where MOSK nodes connected to the external network are located. Those are the nodes where MetalLB speaker components are configured to announce IP addresses for Kubernetes load-balanced services. Mirantis recommends that you use OpenStack controller nodes for this purpose.

The subnets are used to assign addresses to the external interfaces of the MOSK controller nodes and will be used to assign the default gateway to these hosts. The default gateway for other hosts of the MOSK cluster is assigned using the LCM and optionally API/LCM subnets.

Example mosk-racks-external-subnets.yaml

Example of a subnet where a single VLAN segment is stretched to all MOSK controller nodes:

apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: k8s-external
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    k8s-external: true
spec:
  cidr: 10.20.120.0/24
  gateway: 10.20.120.1 # This will be the default gateway on hosts
  includeRanges:
    - 10.20.120.16-10.20.120.20
  nameservers:
    - 8.8.8.8

Example of subnets where separate VLAN segments per rack are used:

apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-k8s-ext
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-k8s-ext: true
spec:
  cidr: 10.20.121.0/24
  gateway: 10.20.121.1 # This will be the default gateway on hosts
  includeRanges:
    - 10.20.121.16-10.20.121.20
  nameservers:
    - 8.8.8.8
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-k8s-ext
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-k8s-ext: true
spec:
  cidr: 10.20.122.0/24
  gateway: 10.20.122.1 # This will be the default gateway on hosts
  includeRanges:
    - 10.20.122.16-10.20.122.20
  nameservers:
    - 8.8.8.8
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-k8s-ext
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-k8s-ext: true
spec:
  cidr: 10.20.123.0/24
  gateway: 10.20.123.1 # This will be the default gateway on hosts
  includeRanges:
    - 10.20.123.16-10.20.123.20
  nameservers:
    - 8.8.8.8

See also

Configure MetalLB

Ceph public network

This network may have per-rack VLANs and IP subnets. The addresses from the subnets are assigned to all MOSK cluster nodes besides Kubernetes manager nodes.

Example mosk-racks-ceph-public-subnets.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-ceph-public
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-ceph-public: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-ceph-public: true
spec:
  cidr: 10.20.131.0/24
  gateway: 10.20.131.1
  includeRanges:
    - 10.20.131.16-10.20.131.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-ceph-public
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-ceph-public: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-ceph-public: true
spec:
  cidr: 10.20.132.0/24
  gateway: 10.20.132.1
  includeRanges:
    - 10.20.132.16-10.20.132.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-ceph-public
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-ceph-public: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-ceph-public: true
spec:
  cidr: 10.20.133.0/24
  gateway: 10.20.133.1
  includeRanges:
    - 10.20.133.16-10.20.133.255
---
# Add more Subnet object templates as required using the above example
# (one subnet per rack)

Ceph cluster network

This network may have per-rack VLANs and IP subnets. The addresses from the subnets are assigned to storage nodes in the MOSK cluster.

Example mosk-racks-ceph-cluster-subnets.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-ceph-cluster
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-ceph-cluster: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-ceph-cluster: true
spec:
  cidr: 10.20.141.0/24
  gateway: 10.20.141.1
  includeRanges:
    - 10.20.141.16-10.20.141.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-ceph-cluster
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-ceph-cluster: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-ceph-cluster: true
spec:
  cidr: 10.20.142.0/24
  gateway: 10.20.142.1
  includeRanges:
    - 10.20.142.16-10.20.142.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-ceph-cluster
  namespace: mosk-namespace-name
  labels:
    ipam/SVC-ceph-cluster: "1"
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-ceph-cluster: true
spec:
  cidr: 10.20.143.0/24
  gateway: 10.20.143.1
  includeRanges:
    - 10.20.143.16-10.20.143.255
---
# Add more Subnet object templates as required using the above example
# (one subnet per rack)

Kubernetes workloads network

This network may include multiple per-rack VLANs and IP subnets. The addresses from the subnets are assigned to all MOSK cluster nodes. For details, see Network types.

Example mosk-racks-k8s-pods.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-k8s-pods
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-k8s-pods: true
spec:
  cidr: 10.20.151.0/24
  gateway: 10.20.151.1
  includeRanges:
    - 10.20.151.16-10.20.151.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-k8s-pods
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-k8s-pods: true
spec:
  cidr: 10.20.152.0/24
  gateway: 10.20.152.1
  includeRanges:
    - 10.20.152.16-10.20.152.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-k8s-pods
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-k8s-pods: true
spec:
  cidr: 10.20.153.0/24
  gateway: 10.20.153.1
  includeRanges:
    - 10.20.153.16-10.20.153.255
---
# Add more Subnet object templates as required using the above example
# (one subnet per rack)

Overlay networks

The underlay network for VXLAN tunnels for the MOSK tenants traffic. If deployed with Tungsten Fabric, it is used for MPLS over UDP+GRE traffic.

Example mosk-racks-tenant-tunnel.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-tenant-tunnel
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-tenant-tunnel: true
spec:
  cidr: 10.20.161.0/24
  gateway: 10.20.161.1
  includeRanges:
    - 10.20.161.16-10.20.161.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-tenant-tunnel
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-tenant-tunnel: true
spec:
  cidr: 10.20.162.0/24
  gateway: 10.20.162.1
  includeRanges:
    - 10.20.162.16-10.20.162.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-tenant-tunnel
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-tenant-tunnel: true
spec:
  cidr: 10.20.163.0/24
  gateway: 10.20.163.1
  includeRanges:
    - 10.20.163.16-10.20.163.255
---
# Add more Subnet object templates as required using the above example
# (one subnet per rack)

Live migration network

The network is used by the Compute service (OpenStack Nova) to transfer data during live migration. Depending on the cloud needs, it can be placed on a dedicated physical network not to affect other networks during live migration.

Example mosk-racks-live-migration.yaml
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-1-live-migration
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-1-live-migration: true
spec:
  cidr: 10.20.171.0/24
  gateway: 10.20.171.1
  includeRanges:
    - 10.20.171.16-10.20.171.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-2-live-migration
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-2-live-migration: true
spec:
  cidr: 10.20.172.0/24
  gateway: 10.20.172.1
  includeRanges:
    - 10.20.172.16-10.20.172.255
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
  name: rack-3-live-migration
  namespace: mosk-namespace-name
  labels:
    kaas.mirantis.com/provider: baremetal
    cluster.sigs.k8s.io/cluster-name: mosk-cluster-name
    rack-3-live-migration: true
spec:
  cidr: 10.20.173.0/24
  gateway: 10.20.173.1
  includeRanges:
    - 10.20.173.16-10.20.173.255
---
# Add more Subnet object templates as required using the above example
# (one subnet per rack)