Add more racks to an existing MOSK cluster¶
This section describes exemplary L2 templates to demonstrate how to add more racks to an existing MOSK cluster.
The following exemplary L2 template belongs to a single-rack MOSK cluster. This template has the following characteristics:
Describes all networks that are used for cluster nodes communication
Can be transformed into several L2 templates depending on nodes roles
Uses the IP gateway in the external network as default route on the nodes in the MOSK cluster
Example of an L2 template for a single-rack cluster
l3Layout:
- subnetName: kaas-mgmt
scope: global
labelSelector:
kaas.mirantis.com/provider: baremetal
kaas-mgmt-subnet: ""
- subnetName: k8s-lcm
scope: namespace
- subnetName: k8s-ext-ipam
scope: namespace
- subnetName: tenant
scope: namespace
- subnetName: k8s-pods
scope: namespace
- subnetName: ceph-front
scope: namespace
- subnetName: ceph-back
scope: namespace
npTemplate: |-
version: 2
ethernets:
{{nic 0}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 0}}
set-name: {{nic 0}}
mtu: 1500
{{nic 1}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 1}}
set-name: {{nic 1}}
mtu: 1500
{{nic 2}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 2}}
set-name: {{nic 2}}
mtu: 9050
{{nic 3}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 3}}
set-name: {{nic 3}}
mtu: 9050
bonds:
bond0:
interfaces:
- {{nic 0}}
- {{nic 1}}
parameters:
mode: 802.3ad
transmit-hash-policy: layer3+4
mtu: 1500
bond1:
interfaces:
- {{nic 2}}
- {{nic 3}}
parameters:
mode: 802.3ad
transmit-hash-policy: layer3+4
mtu: 9050
vlans:
k8s-lcm-v:
id: 738
link: bond0
k8s-pod-v:
id: 731
link: bond1
mtu: 9000
k8s-ext-v:
id: 736
link: bond1
mtu: 9000
tenant-vlan:
id: 732
link: bond1
addresses:
- {{ip "tenant-vlan:tenant"}}
mtu: 9050
ceph-front-v:
id: 733
link: bond1
addresses:
- {{ip "ceph-front-v:ceph-front"}}
mtu: 9000
ceph-back-v:
id: 734
link: bond1
addresses:
- {{ip "ceph-back-v:ceph-back"}}
mtu: 9000
bridges:
k8s-lcm:
interfaces: [k8s-lcm-v]
addresses:
- {{ip "k8s-lcm:k8s-lcm"}}
routes:
# to management network of MCC cluster
- to: {{cidr_from_subnet "kaas-mgmt"}}
via: {{gateway_from_subnet "k8s-lcm"}}
table: 101
# fips network
- to: 10.159.156.0/22
via: {{gateway_from_subnet "k8s-lcm"}}
table: 101
routing-policy:
- from: {{cidr_from_subnet "k8s-lcm"}}
table: 101
k8s-pods:
interfaces: [k8s-pod-v]
addresses:
- {{ip "k8s-pods:k8s-pods"}}
mtu: 9000
k8s-ext:
interfaces: [k8s-ext-v]
addresses:
- {{ip "k8s-ext:k8s-ext-ipam"}}
gateway4: {{gateway_from_subnet "k8s-ext-ipam"}}
nameservers:
addresses: {{nameservers_from_subnet "k8s-ext-ipam"}}
mtu: 9000
## FIP Bridge
br-fip:
interfaces: [bond1]
mtu: 9050
To add nodes to the new rack of the same cluster:
Create
Subnetobjects for the following networks: LCM, workload, tenant, and Ceph (where applicable).Create a new L2 template that nodes in a new rack will use.
In this template, configure the external network to be either stretched between racks or connected to the first rack only.
Caution
API/LCM network is the first rack LCM network in our example, since a single-rack MOSK cluster was deployed first. Therefore, only the first rack can contain Kubernetes master nodes that provide access to Kubernetes API.
In the L2 template for the first rack, add IP routes pointing to the networks in the new rack.
The following examples contain:
The modified L2 template for the first rack. Routes added to the second rack are highlighted.
The new L2 template for the second rack with external network that is stretched between racks. The IP gateway in the external network is used as the default route on the nodes of the second rack.
Example of a modified L2 template for the first rack with routes
to the second rack
l3Layout:
- subnetName: kaas-mgmt
scope: global
labelSelector:
kaas.mirantis.com/provider: baremetal
kaas-mgmt-subnet: ""
- subnetName: k8s-lcm
scope: namespace
- subnetName: k8s-ext-ipam
scope: namespace
- subnetName: tenant
scope: namespace
- subnetName: k8s-pods
scope: namespace
- subnetName: ceph-front
scope: namespace
- subnetName: ceph-back
scope: namespace
- subnetName: k8s-lcm-rack2
scope: namespace
- subnetName: tenant-rack2
scope: namespace
- subnetName: k8s-pods-rack2
scope: namespace
- subnetName: ceph-front-rack2
scope: namespace
- subnetName: ceph-back-rack2
scope: namespace
npTemplate: |-
version: 2
ethernets:
{{nic 0}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 0}}
set-name: {{nic 0}}
mtu: 1500
{{nic 1}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 1}}
set-name: {{nic 1}}
mtu: 1500
{{nic 2}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 2}}
set-name: {{nic 2}}
mtu: 9050
{{nic 3}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 3}}
set-name: {{nic 3}}
mtu: 9050
bonds:
bond0:
interfaces:
- {{nic 0}}
- {{nic 1}}
parameters:
mode: 802.3ad
transmit-hash-policy: layer3+4
mtu: 1500
bond1:
interfaces:
- {{nic 2}}
- {{nic 3}}
parameters:
mode: 802.3ad
transmit-hash-policy: layer3+4
mtu: 9050
vlans:
k8s-lcm-v:
id: 738
link: bond0
k8s-pod-v:
id: 731
link: bond1
mtu: 9000
k8s-ext-v:
id: 736
link: bond1
mtu: 9000
tenant-vlan:
id: 732
link: bond1
addresses:
- {{ip "tenant-vlan:tenant"}}
routes:
# to 2nd rack of MOSK cluster
- to: {{cidr_from_subnet "tenant-rack2"}}
via: {{gateway_from_subnet "tenant"}}
mtu: 9050
ceph-front-v:
id: 733
link: bond1
addresses:
- {{ip "ceph-front-v:ceph-front"}}
routes:
# to 2nd rack of MOSK cluster
- to: {{cidr_from_subnet "ceph-front-rack2"}}
via: {{gateway_from_subnet "ceph-front"}}
mtu: 9000
ceph-back-v:
id: 734
link: bond1
addresses:
- {{ip "ceph-back-v:ceph-back"}}
routes:
# to 2nd rack of MOSK cluster
- to: {{cidr_from_subnet "ceph-back-rack2"}}
via: {{gateway_from_subnet "ceph-back"}}
mtu: 9000
bridges:
k8s-lcm:
interfaces: [k8s-lcm-v]
addresses:
- {{ip "k8s-lcm:k8s-lcm"}}
nameservers:
addresses: {{nameservers_from_subnet "k8s-lcm"}}
routes:
# to management network of Container Cloud cluster
- to: {{cidr_from_subnet "kaas-mgmt"}}
via: {{gateway_from_subnet "k8s-lcm"}}
table: 101
# fips network
- to: 10.159.156.0/22
via: {{gateway_from_subnet "k8s-lcm"}}
table: 101
# to 2nd rack of MOSK cluster
- to: {{cidr_from_subnet "k8s-lcm-rack2"}}
via: {{gateway_from_subnet "k8s-lcm"}}
table: 101
routing-policy:
- from: {{cidr_from_subnet "k8s-lcm"}}
table: 101
k8s-pods:
interfaces: [k8s-pod-v]
addresses:
- {{ip "k8s-pods:k8s-pods"}}
routes:
# to 2nd rack of MOSK cluster
- to: {{cidr_from_subnet "k8s-pods-rack2"}}
via: {{gateway_from_subnet "k8s-pods"}}
mtu: 9000
k8s-ext:
interfaces: [k8s-ext-v]
addresses:
- {{ip "k8s-ext:k8s-ext-ipam"}}
gateway4: {{gateway_from_subnet "k8s-ext-ipam"}}
nameservers:
addresses: {{nameservers_from_subnet "k8s-ext-ipam"}}
mtu: 9000
## FIP Bridge
br-fip:
interfaces: [bond1]
mtu: 9050
Example of a new L2 template for the second rack with external
network
l3Layout: - subnetName: kaas-mgmt scope: global labelSelector: kaas.mirantis.com/provider: baremetal kaas-mgmt-subnet: "" - subnetName: k8s-lcm scope: namespace - subnetName: k8s-ext-ipam scope: namespace - subnetName: tenant scope: namespace - subnetName: k8s-pods scope: namespace - subnetName: ceph-front scope: namespace - subnetName: ceph-back scope: namespace - subnetName: k8s-lcm-rack2 scope: namespace - subnetName: tenant-rack2 scope: namespace - subnetName: k8s-pods-rack2 scope: namespace - subnetName: ceph-front-rack2 scope: namespace - subnetName: ceph-back-rack2 scope: namespace npTemplate: |- version: 2 ethernets: {{nic 0}}: dhcp4: false dhcp6: false match: macaddress: {{mac 0}} set-name: {{nic 0}} mtu: 1500 {{nic 1}}: dhcp4: false dhcp6: false match: macaddress: {{mac 1}} set-name: {{nic 1}} mtu: 1500 {{nic 2}}: dhcp4: false dhcp6: false match: macaddress: {{mac 2}} set-name: {{nic 2}} mtu: 9050 {{nic 3}}: dhcp4: false dhcp6: false match: macaddress: {{mac 3}} set-name: {{nic 3}} mtu: 9050 bonds: bond0: interfaces: - {{nic 0}} - {{nic 1}} parameters: mode: 802.3ad transmit-hash-policy: layer3+4 mtu: 1500 bond1: interfaces: - {{nic 2}} - {{nic 3}} parameters: mode: 802.3ad transmit-hash-policy: layer3+4 mtu: 9050 vlans: k8s-lcm-v: id: 738 link: bond0 k8s-pod-v: id: 731 link: bond1 mtu: 9000 k8s-ext-v: id: 736 link: bond1 mtu: 9000 tenant-vlan: id: 732 link: bond1 addresses: - {{ip "tenant-vlan:tenant-rack2"}} routes: # to 2nd rack of MOSK cluster - to: {{cidr_from_subnet "tenant"}} via: {{gateway_from_subnet "tenant-rack2"}} mtu: 9050 ceph-front-v: id: 733 link: bond1 addresses: - {{ip "ceph-front-v:ceph-front-rack2"}} routes: # to 1st rack of MOSK cluster - to: {{cidr_from_subnet "ceph-front"}} via: {{gateway_from_subnet "ceph-front-rack2"}} mtu: 9000 ceph-back-v: id: 734 link: bond1 addresses: - {{ip "ceph-back-v:ceph-back-rack2"}} routes: # to 2nd rack of MOSK cluster - to: {{cidr_from_subnet "ceph-back"}} via: {{gateway_from_subnet "ceph-back-rack2"}} mtu: 9000 bridges: k8s-lcm: interfaces: [k8s-lcm-v] addresses: - {{ip "k8s-lcm:k8s-lcm-rack2"}} nameservers: addresses: {{nameservers_from_subnet "k8s-lcm-rack2"}} routes: # to management network of Container Cloud cluster - to: {{cidr_from_subnet "kaas-mgmt"}} via: {{gateway_from_subnet "k8s-lcm-rack2"}} table: 101 # fips network - to: 10.159.156.0/22 via: {{gateway_from_subnet "k8s-lcm-rack2"}} table: 101 # to API/LCM network of MOSK cluster - to: {{cidr_from_subnet "k8s-lcm"}} via: {{gateway_from_subnet "k8s-lcm-rack2"}} table: 101 routing-policy: - from: {{cidr_from_subnet "k8s-lcm-rack2"}} table: 101 k8s-pods: interfaces: [k8s-pod-v] addresses: - {{ip "k8s-pods:k8s-pods-rack2"}} routes: # to 2nd rack of MOSK cluster - to: {{cidr_from_subnet "k8s-pods"}} via: {{gateway_from_subnet "k8s-pods-rack2"}} mtu: 9000 k8s-ext: interfaces: [k8s-ext-v] addresses: - {{ip "k8s-ext:k8s-ext-ipam"}} gateway4: {{gateway_from_subnet "k8s-ext-ipam"}} nameservers: addresses: {{nameservers_from_subnet "k8s-ext-ipam"}} mtu: 9000 ## FIP Bridge br-fip: interfaces: [bond1] mtu: 9050