Add more racks to an existing MOSK cluster

This section describes exemplary L2 templates to demonstrate how to add more racks to an existing MOSK cluster.

The following exemplary L2 template belongs to a single-rack MOSK cluster. This template has the following characteristics:

  • Describes all networks that are used for cluster nodes communication

  • Can be transformed into several L2 templates depending on nodes roles

  • Uses the IP gateway in the external network as default route on the nodes in the MOSK cluster

Example of an L2 template for a single-rack cluster
l3Layout:
- subnetName: kaas-mgmt
  scope: global
  labelSelector:
    kaas.mirantis.com/provider: baremetal
    kaas-mgmt-subnet: ""
- subnetName: k8s-lcm
  scope: namespace
- subnetName: k8s-ext-ipam
  scope: namespace
- subnetName: tenant
  scope: namespace
- subnetName: k8s-pods
  scope: namespace
- subnetName: ceph-front
  scope: namespace
- subnetName: ceph-back
  scope: namespace
npTemplate: |-
  version: 2
  ethernets:
    {{nic 0}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 0}}
      set-name: {{nic 0}}
      mtu: 1500
    {{nic 1}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 1}}
      set-name: {{nic 1}}
      mtu: 1500
    {{nic 2}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 2}}
      set-name: {{nic 2}}
      mtu: 9050
    {{nic 3}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 3}}
      set-name: {{nic 3}}
      mtu: 9050
  bonds:
    bond0:
      interfaces:
        - {{nic 0}}
        - {{nic 1}}
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
      mtu: 1500
    bond1:
      interfaces:
        - {{nic 2}}
        - {{nic 3}}
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
      mtu: 9050
  vlans:
    k8s-lcm-v:
      id: 738
      link: bond0
    k8s-pod-v:
      id: 731
      link: bond1
      mtu: 9000
    k8s-ext-v:
      id: 736
      link: bond1
      mtu: 9000
    tenant-vlan:
      id: 732
      link: bond1
      addresses:
        - {{ip "tenant-vlan:tenant"}}
      mtu: 9050
    ceph-front-v:
      id: 733
      link: bond1
      addresses:
        - {{ip "ceph-front-v:ceph-front"}}
      mtu: 9000
    ceph-back-v:
      id: 734
      link: bond1
      addresses:
        - {{ip "ceph-back-v:ceph-back"}}
      mtu: 9000
  bridges:
    k8s-lcm:
      interfaces: [k8s-lcm-v]
      addresses:
        - {{ip "k8s-lcm:k8s-lcm"}}
      routes:
        # to management network of MCC cluster
        - to: {{cidr_from_subnet "kaas-mgmt"}}
          via: {{gateway_from_subnet "k8s-lcm"}}
          table: 101
        # fips network
        - to: 10.159.156.0/22
          via: {{gateway_from_subnet "k8s-lcm"}}
          table: 101
      routing-policy:
        - from: {{cidr_from_subnet "k8s-lcm"}}
          table: 101
    k8s-pods:
      interfaces: [k8s-pod-v]
      addresses:
        - {{ip "k8s-pods:k8s-pods"}}
      mtu: 9000
    k8s-ext:
      interfaces: [k8s-ext-v]
      addresses:
        - {{ip "k8s-ext:k8s-ext-ipam"}}
      gateway4: {{gateway_from_subnet "k8s-ext-ipam"}}
      nameservers:
        addresses: {{nameservers_from_subnet "k8s-ext-ipam"}}
      mtu: 9000
    ## FIP Bridge
    br-fip:
      interfaces: [bond1]
      mtu: 9050

To add nodes to the new rack of the same cluster:

  1. Create Subnet objects for the following networks: LCM, workload, tenant, and Ceph (where applicable).

  2. Create a new L2 template that nodes in a new rack will use.

  3. In this template, configure the external network to be either stretched between racks or connected to the first rack only.

    Caution

    API/LCM network is the first rack LCM network in our example, since a single-rack MOSK cluster was deployed first. Therefore, only the first rack can contain Kubernetes master nodes that provide access to Kubernetes API.

  4. In the L2 template for the first rack, add IP routes pointing to the networks in the new rack.

The following examples contain:

  • The modified L2 template for the first rack. Routes added to the second rack are highlighted.

  • The new L2 template for the second rack with external network that is stretched between racks. The IP gateway in the external network is used as the default route on the nodes of the second rack.

Example of a modified L2 template for the first rack with routes to the second rack
l3Layout:
- subnetName: kaas-mgmt
  scope: global
  labelSelector:
    kaas.mirantis.com/provider: baremetal
    kaas-mgmt-subnet: ""
- subnetName: k8s-lcm
  scope: namespace
- subnetName: k8s-ext-ipam
  scope: namespace
- subnetName: tenant
  scope: namespace
- subnetName: k8s-pods
  scope: namespace
- subnetName: ceph-front
  scope: namespace
- subnetName: ceph-back
  scope: namespace
- subnetName: k8s-lcm-rack2
  scope: namespace
- subnetName: tenant-rack2
  scope: namespace
- subnetName: k8s-pods-rack2
  scope: namespace
- subnetName: ceph-front-rack2
  scope: namespace
- subnetName: ceph-back-rack2
  scope: namespace
npTemplate: |-
  version: 2
  ethernets:
    {{nic 0}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 0}}
      set-name: {{nic 0}}
      mtu: 1500
    {{nic 1}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 1}}
      set-name: {{nic 1}}
      mtu: 1500
    {{nic 2}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 2}}
      set-name: {{nic 2}}
      mtu: 9050
    {{nic 3}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 3}}
      set-name: {{nic 3}}
      mtu: 9050
  bonds:
    bond0:
      interfaces:
        - {{nic 0}}
        - {{nic 1}}
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
      mtu: 1500
    bond1:
      interfaces:
        - {{nic 2}}
        - {{nic 3}}
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
      mtu: 9050
  vlans:
    k8s-lcm-v:
      id: 738
      link: bond0
    k8s-pod-v:
      id: 731
      link: bond1
      mtu: 9000
    k8s-ext-v:
      id: 736
      link: bond1
      mtu: 9000
    tenant-vlan:
      id: 732
      link: bond1
      addresses:
        - {{ip "tenant-vlan:tenant"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "tenant-rack2"}}
          via: {{gateway_from_subnet "tenant"}}
      mtu: 9050
    ceph-front-v:
      id: 733
      link: bond1
      addresses:
        - {{ip "ceph-front-v:ceph-front"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "ceph-front-rack2"}}
          via: {{gateway_from_subnet "ceph-front"}}
      mtu: 9000
    ceph-back-v:
      id: 734
      link: bond1
      addresses:
        - {{ip "ceph-back-v:ceph-back"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "ceph-back-rack2"}}
          via: {{gateway_from_subnet "ceph-back"}}
      mtu: 9000
  bridges:
    k8s-lcm:
      interfaces: [k8s-lcm-v]
      addresses:
        - {{ip "k8s-lcm:k8s-lcm"}}
      nameservers:
        addresses: {{nameservers_from_subnet "k8s-lcm"}}
      routes:
        # to management network of Container Cloud cluster
        - to: {{cidr_from_subnet "kaas-mgmt"}}
          via: {{gateway_from_subnet "k8s-lcm"}}
          table: 101
        # fips network
        - to: 10.159.156.0/22
          via: {{gateway_from_subnet "k8s-lcm"}}
          table: 101
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "k8s-lcm-rack2"}}
          via: {{gateway_from_subnet "k8s-lcm"}}
          table: 101
      routing-policy:
        - from: {{cidr_from_subnet "k8s-lcm"}}
          table: 101
    k8s-pods:
      interfaces: [k8s-pod-v]
      addresses:
        - {{ip "k8s-pods:k8s-pods"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "k8s-pods-rack2"}}
          via: {{gateway_from_subnet "k8s-pods"}}
      mtu: 9000
    k8s-ext:
      interfaces: [k8s-ext-v]
      addresses:
        - {{ip "k8s-ext:k8s-ext-ipam"}}
      gateway4: {{gateway_from_subnet "k8s-ext-ipam"}}
      nameservers:
        addresses: {{nameservers_from_subnet "k8s-ext-ipam"}}
      mtu: 9000
    ## FIP Bridge
    br-fip:
      interfaces: [bond1]
      mtu: 9050
Example of a new L2 template for the second rack with external network
l3Layout:
- subnetName: kaas-mgmt
  scope: global
  labelSelector:
    kaas.mirantis.com/provider: baremetal
    kaas-mgmt-subnet: ""
- subnetName: k8s-lcm
  scope: namespace
- subnetName: k8s-ext-ipam
  scope: namespace
- subnetName: tenant
  scope: namespace
- subnetName: k8s-pods
  scope: namespace
- subnetName: ceph-front
  scope: namespace
- subnetName: ceph-back
  scope: namespace
- subnetName: k8s-lcm-rack2
  scope: namespace
- subnetName: tenant-rack2
  scope: namespace
- subnetName: k8s-pods-rack2
  scope: namespace
- subnetName: ceph-front-rack2
  scope: namespace
- subnetName: ceph-back-rack2
  scope: namespace
npTemplate: |-
  version: 2
  ethernets:
    {{nic 0}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 0}}
      set-name: {{nic 0}}
      mtu: 1500
    {{nic 1}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 1}}
      set-name: {{nic 1}}
      mtu: 1500
    {{nic 2}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 2}}
      set-name: {{nic 2}}
      mtu: 9050
    {{nic 3}}:
      dhcp4: false
      dhcp6: false
      match:
        macaddress: {{mac 3}}
      set-name: {{nic 3}}
      mtu: 9050
  bonds:
    bond0:
      interfaces:
        - {{nic 0}}
        - {{nic 1}}
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
      mtu: 1500
    bond1:
      interfaces:
        - {{nic 2}}
        - {{nic 3}}
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
      mtu: 9050
  vlans:
    k8s-lcm-v:
      id: 738
      link: bond0
    k8s-pod-v:
      id: 731
      link: bond1
      mtu: 9000
    k8s-ext-v:
      id: 736
      link: bond1
      mtu: 9000
    tenant-vlan:
      id: 732
      link: bond1
      addresses:
        - {{ip "tenant-vlan:tenant-rack2"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "tenant"}}
          via: {{gateway_from_subnet "tenant-rack2"}}
      mtu: 9050
    ceph-front-v:
      id: 733
      link: bond1
      addresses:
        - {{ip "ceph-front-v:ceph-front-rack2"}}
      routes:
        # to 1st rack of MOSK cluster
        - to: {{cidr_from_subnet "ceph-front"}}
          via: {{gateway_from_subnet "ceph-front-rack2"}}
      mtu: 9000
    ceph-back-v:
      id: 734
      link: bond1
      addresses:
        - {{ip "ceph-back-v:ceph-back-rack2"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "ceph-back"}}
          via: {{gateway_from_subnet "ceph-back-rack2"}}
      mtu: 9000
  bridges:
    k8s-lcm:
      interfaces: [k8s-lcm-v]
      addresses:
        - {{ip "k8s-lcm:k8s-lcm-rack2"}}
      nameservers:
        addresses: {{nameservers_from_subnet "k8s-lcm-rack2"}}
      routes:
        # to management network of Container Cloud cluster
        - to: {{cidr_from_subnet "kaas-mgmt"}}
          via: {{gateway_from_subnet "k8s-lcm-rack2"}}
          table: 101
        # fips network
        - to: 10.159.156.0/22
          via: {{gateway_from_subnet "k8s-lcm-rack2"}}
          table: 101
        # to API/LCM network of MOSK cluster
        - to: {{cidr_from_subnet "k8s-lcm"}}
          via: {{gateway_from_subnet "k8s-lcm-rack2"}}
          table: 101
      routing-policy:
        - from: {{cidr_from_subnet "k8s-lcm-rack2"}}
          table: 101
    k8s-pods:
      interfaces: [k8s-pod-v]
      addresses:
        - {{ip "k8s-pods:k8s-pods-rack2"}}
      routes:
        # to 2nd rack of MOSK cluster
        - to: {{cidr_from_subnet "k8s-pods"}}
          via: {{gateway_from_subnet "k8s-pods-rack2"}}
      mtu: 9000
    k8s-ext:
      interfaces: [k8s-ext-v]
      addresses:
        - {{ip "k8s-ext:k8s-ext-ipam"}}
      gateway4: {{gateway_from_subnet "k8s-ext-ipam"}}
      nameservers:
        addresses: {{nameservers_from_subnet "k8s-ext-ipam"}}
      mtu: 9000
    ## FIP Bridge
    br-fip:
      interfaces: [bond1]
      mtu: 9050