Create subnets for a managed cluster using CLI¶
After creating the MetalLB configuration as described in Configure MetalLB and before creating an L2 template, create the required subnets to use in the
L2 template to allocate IP addresses for the managed cluster nodes.
To create subnets for a managed cluster using CLI:
Log in to a local machine where your management cluster
kubeconfig
is located and wherekubectl
is installed.Note
The management cluster
kubeconfig
is created during the last stage of the management cluster bootstrap.Create a cluster using one of the following options:
Container Cloud web UI: Create a cluster using web UI
Container Cloud API: Cluster resource
Create the
subnet.yaml
file with a number of global or namespaced subnets depending on the configuration of your cluster:kubectl --kubeconfig <pathToManagementClusterKubeconfig> apply -f <SubnetFileName.yaml>
Note
In the command above and in the steps below, substitute the parameters enclosed in angle brackets with the corresponding values.
Example of a
subnet.yaml
file:apiVersion: ipam.mirantis.com/v1alpha1 kind: Subnet metadata: name: demo namespace: demo-namespace labels: kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one spec: cidr: 10.11.0.0/24 gateway: 10.11.0.9 includeRanges: - 10.11.0.5-10.11.0.70 nameservers: - 172.18.176.6
Note
The
kaas.mirantis.com/region
label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.¶ Parameter
Description
cidr
(singular)A valid IPv4 CIDR, for example, 10.11.0.0/24.
includeRanges
(list)A comma-separated list of IP address ranges within the given CIDR that should be used in the allocation of IPs for nodes. The gateway, network, broadcast, and DNSaddresses will be excluded (protected) automatically if they intersect with one of the range. The IPs outside the given ranges will not be used in the allocation. Each element of the list can be either an interval
10.11.0.5-10.11.0.70
or a single address10.11.0.77
. TheincludeRanges
parameter is mutually exclusive withexcludeRanges
.excludeRanges
(list)A comma-separated list of IP address ranges within the given CIDR that should not be used in the allocation of IPs for nodes. The IPs within the given CIDR but outside the given ranges will be used in the allocation. The gateway, network, broadcast, and DNS addresses will be excluded (protected) automatically if they are included in the CIDR. Each element of the list can be either an interval
10.11.0.5-10.11.0.70
or a single address10.11.0.77
. TheexcludeRanges
parameter is mutually exclusive withincludeRanges
.useWholeCidr
(boolean)If set to
true
, the subnet address (10.11.0.0 in the example above) and the broadcast address (10.11.0.255 in the example above) are included into the address allocation for nodes. Otherwise, (false
by default), the subnet address and broadcast address will be excluded from the address allocation.gateway
(singular)A valid gateway address, for example, 10.11.0.9.
nameservers
(list)A list of the IP addresses of name servers. Each element of the list is a single address, for example, 172.18.176.6.
Caution
The subnet for the PXE network of the management cluster is automatically created during deployment.
The subnet for the LCM network must contain the
ipam/SVC-k8s-lcm: "1"
label. For details, see Service labels and their life cycle.Each cluster must use at least one subnet for its LCM network. Every node must have the address allocated in the LCM network using such subnet(s).
Each node of every cluster must have only one IP address in the LCM network that is allocated from one of the
Subnet
objects having theipam/SVC-k8s-lcm
label defined. Therefore, allSubnet
objects used for LCM networks must have theipam/SVC-k8s-lcm
label defined. For details, see Service labels and their life cycle.Note
You may use different subnets to allocate IP addresses to different Container Cloud components in your cluster. Add a label with the
ipam/SVC-
prefix to each subnet that is used to configure a Container Cloud service. For details, see Service labels and their life cycle and the optional steps below.Caution
Use of a dedicated network for Kubernetes pods traffic, for external connection to the Kubernetes services exposed by the cluster, and for the Ceph cluster access and replication traffic is available as Technology Preview. Use such configurations for testing and evaluation purposes only. For the Technology Preview feature definition, refer to Technology Preview features.
Optional. Technology Preview. Add a subnet for the externally accessible API endpoint of the managed cluster.
Make sure that
loadBalancerHost
is set to""
(empty string) in theCluster
spec.spec: providerSpec: value: apiVersion: baremetal.k8s.io/v1alpha1 kind: BaremetalClusterProviderSpec ... loadBalancerHost: ""
Create a subnet with the
ipam/SVC-LBhost
label having the"1"
value to make thebaremetal-provider
use this subnet for allocation of cluster API endpoints addresses.
One IP address will be allocated for each cluster to serve its Kubernetes/MKE API endpoint.
Caution
Make sure that master nodes have host local-link addresses in the same subnet as the cluster API endpoint address. These host IP addresses will be used for VRRP traffic. The cluster API endpoint address will be assigned to the same interface on one of the master nodes where these host IPs are assigned.
Note
We highly recommend that you assign the cluster API endpoint address from the LCM network. For details on cluster networks types, refer to Managed cluster networking. See also the Single managed cluster use case example in the following table.
You can use several options of addresses allocation scope of API endpoints using subnets:
Use case
Example configuration
Several managed clusters in one management cluster
Create a subnet in the
default
namespace with no reference to any cluster.apiVersion: ipam.mirantis.com/v1alpha1 kind: Subnet metadata: name: lbhost-per-region namespace: default labels: kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one ipam/SVC-LBhost: "1" spec: cidr: 191.11.0.0/24 includeRanges: - 191.11.0.6-191.11.0.20
Note
The
kaas.mirantis.com/region
label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.Warning
Combining the
ipam/SVC-LBhost
label with any other service labels on a single subnet is not supported. Use a dedicated subnet for addresses allocation for cluster API endpoints.Several managed clusters in a project
Create a subnet in a namespace corresponding to your project with no reference to any cluster. Such subnet has priority over the one described above.
apiVersion: ipam.mirantis.com/v1alpha1 kind: Subnet metadata: name: lbhost-per-namespace namespace: my-project labels: kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one ipam/SVC-LBhost: "1" spec: cidr: 191.11.0.0/24 includeRanges: - 191.11.0.6-191.11.0.20
Warning
Combining the
ipam/SVC-LBhost
label with any other service labels on a single subnet is not supported. Use a dedicated subnet for addresses allocation for cluster API endpoints.Single managed cluster
Create a subnet in a namespace corresponding to your project with a reference to the target cluster using the
cluster.sigs.k8s.io/cluster-name
label. Such subnet has priority over the ones described above. In this case, it is not obligatory to use a dedicated subnet for addresses allocation of API endpoints. You can add theipam/SVC-LBhost
label to the LCM subnet, and one of the addresses from this subnet will be allocated for an API endpoint:apiVersion: ipam.mirantis.com/v1alpha1 kind: Subnet metadata: name: lbhost-per-cluster namespace: my-project labels: kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one ipam/SVC-LBhost: "1" ipam/SVC-k8s-lcm: "1" cluster.sigs.k8s.io/cluster-name: my-cluster spec: cidr: 10.11.0.0/24 includeRanges: - 10.11.0.6-10.11.0.50
Warning
You can combine the
ipam/SVC-LBhost
label only with the following service labels on a single subnet:ipam/SVC-k8s-lcm
ipam/SVC-ceph-cluster
ipam/SVC-ceph-public
Otherwise, use a dedicated subnet for address allocation for the cluster API endpoint. Other combinations are not supported and can lead to unexpected results.
The above options can be used in conjunction. For example, you can define a subnet for a region, a number of subnets within this region defined for particular namespaces, and a number of subnets within the same region and namespaces defined for particular clusters.
Optional. Add a subnet(s) for the Storage access network.
Set the
ipam/SVC-ceph-public
label with the value"1"
to create a subnet that will be used to configure the Ceph public network.Set the
cluster.sigs.k8s.io/cluster-name
label to the name of the target cluster during the subnet creation.Use this subnet in the L2 template for storage nodes.
Assign this subnet to the interface connected to your Storage access network.
Ceph will automatically use this subnet for its external connections.
A Ceph OSD will look for and bind to an address from this subnet when it is started on a machine.
Optional. Add a subnet(s) for the Storage replication network.
Set the
ipam/SVC-ceph-cluster
label with the value"1"
to create a subnet that will be used to configure the Ceph cluster network.Set the
cluster.sigs.k8s.io/cluster-name
label to the name of the target cluster during the subnet creation.Use this subnet in the L2 template for storage nodes.
Assign this subnet to the interface connected to your Storage replication network.
Ceph will automatically use this subnet for its internal replication traffic.
Optional. Add a subnet for Kubernetes pods traffic.
Use this subnet in the L2 template for all nodes in the cluster.
Assign this subnet to the interface connected to your Kubernetes workloads network.
Use the
npTemplate.bridges.k8s-pods
bridge name in the L2 template. This bridge name is reserved for the Kubernetes workloads network. When thek8s-pods
bridge is defined in an L2 template, Calico CNI uses that network for routing the pods traffic between nodes.
Optional. Add subnets for configuring multiple DHCP ranges. For details, see Configure multiple DHCP ranges using Subnet resources.
Verify that the subnet is successfully created:
kubectl get subnet kaas-mgmt -o yaml
In the system output, verify the
status
fields of thesubnet.yaml
file using the table below.¶ Parameter
Description
state
Since 2.23.0Contains a short state description and a more detailed one if applicable. The short status values are as follows:
OK
- object is operational.ERR
- object is non-operational. This status has a detailed description in themessages
list.TERM
- object was deleted and is terminating.
messages
Since 2.23.0Contains error or warning messages if the object state is
ERR
. For example,ERR: Wrong includeRange for CIDR…
.statusMessage
Deprecated since Container Cloud 2.23.0 and will be removed in one of the following releases in favor of
state
andmessages
. Since Container Cloud 2.24.0, this field is not set for the objects of newly created clusters.cidr
Reflects the actual CIDR, has the same meaning as
spec.cidr
.gateway
Reflects the actual gateway, has the same meaning as
spec.gateway
.nameservers
Reflects the actual name servers, has same meaning as
spec.nameservers
.ranges
Specifies the address ranges that are calculated using the fields from
spec: cidr, includeRanges, excludeRanges, gateway, useWholeCidr
. These ranges are directly used for nodes IP allocation.allocatable
Includes the number of currently available IP addresses that can be allocated for nodes from the subnet.
allocatedIPs
Specifies the list of IPv4 addresses with the corresponding
IPaddr
object IDs that were already allocated from the subnet.capacity
Contains the total number of IP addresses being held by ranges that equals to a sum of the
allocatable
andallocatedIPs
parameters values.objCreated
Date, time, and IPAM version of the
Subnet
CR creation.objStatusUpdated
Date, time, and IPAM version of the last update of the
status
field in theSubnet
CR.objUpdated
Date, time, and IPAM version of the last
Subnet
CR update bykaas-ipam
.Example of a successfully created subnet:
apiVersion: ipam.mirantis.com/v1alpha1 kind: Subnet metadata: labels: ipam/UID: 6039758f-23ee-40ba-8c0f-61c01b0ac863 kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one ipam/SVC-k8s-lcm: "1" name: kaas-mgmt namespace: default spec: cidr: 172.16.170.0/24 excludeRanges: - 172.16.170.100 - 172.16.170.101-172.16.170.139 gateway: 172.16.170.1 includeRanges: - 172.16.170.70-172.16.170.99 nameservers: - 172.18.176.6 - 172.18.224.6 status: allocatable: 27 allocatedIPs: - 172.16.170.70:ebabace8-7d9e-4913-a938-3d9e809f49fc - 172.16.170.71:c1109596-fba1-471b-950b-b1b60ef2c37c - 172.16.170.72:94c25734-c046-4a7e-a0fb-75582c5f20a9 capacity: 30 checksums: annotations: sha256:38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed labels: sha256:5ed97704b05f15b204c1347603f9749ac015c29a4a16c6f599eed06babfb312e spec: sha256:60ead7c744564b3bfbbb3c4e846bce54e9128be49a279bf0c2bbebac2cfcebe6 cidr: 172.16.170.0/24 gateway: 172.16.170.1 labelSetChecksum: 5ed97704b05f15b204c1347603f9749ac015c29a4a16c6f599eed06babfb312e nameservers: - 172.18.176.6 - 172.18.224.6 objCreated: 2023-03-03T03:06:20.00000Z by v6.4.999-20230127-091906-c451398 objStatusUpdated: 2023-03-03T04:05:14.48469Z by v6.4.999-20230127-091906-c451398 objUpdated: 2023-03-03T04:05:14.48469Z by v6.4.999-20230127-091906-c451398 ranges: - 172.16.170.70-172.16.170.99 state: OK
Proceed to creating an L2 template for one or multiple managed clusters as described in Create L2 templates.