Deploy a management cluster using the Container Cloud API¶

Available since 2.25.0

This section contains an overview of the cluster-related objects along with the configuration procedure of these objects during deployment of a management cluster using Bootstrap v2 through the Container Cloud API.

Overview of the cluster-related objects in the Container Cloud API/CLI¶

The following cluster-related objects are available through the Container Cloud API. Use these objects to deploy a management cluster using the Container Cloud API.

Cluster objects¶
Object name	Description
`BootstrapRegion`	Region and provider names for a management cluster and all related objects. First object to create in the bootstrap cluster. For the bootstrap region definition, see Introduction.
`ProviderCredentials`	Provider credentials to access cloud infrastructure where the Container Cloud machines are deployed. Requires the region name label.
`SSHKey`	Optional. SSH configuration with any number of SSH public keys to be added to cluster machines. By default, any bootstrap cluster has a pregenerated `bootstrap-key` object to use for the cluster configuration. This is the service SSH key used by the Bootstrap Controller to access machines for their deployment. The private part of `bootstrap-key` is always saved to `kaas-bootstrap/ssh_key`.
`Proxy`	Proxy configuration. Mandatory for offline environments with no direct access to the Internet. Such configuration usually contains proxy for the bootstrap cluster and already has the `bootstrap-proxy` object to use in the cluster configuration by default. For proxy implementation details, see Requirements for a MITM proxy and Proxy and cache support.
`Cluster`	Provider-specific configuration for a management cluster. Requires the region name label with the name of the `BootstrapRegion` object.
`Machine`	Machine configuration that must fit the following requirements: Role - only `manager` Number - odd for the management cluster HA Mandatory labels - `region`, `provider`, `cluster-name`
`ServiceUser`	Service user is the initial user to create in Keycloak for access to a newly deployed management cluster. By default, it has the `global-admin`, `operator` (namespaced), and `bm-pool-operator` (namespaced) roles. You can delete `serviceuser` after setting up other required users with specific roles or after any integration with an external identity provider, such as LDAP.
`VMTemplate`	For the vSphere provider only. A base template that contains the operating system configuration for vSphere virtual machines.
`RHELLicense`	Optional and required for the vSphere RHEL-based clusters only. RHEL license details to be applied to a vSphere VM. Use one of the following set of parameters for RHEL machines subscription: The user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters. The activation key and organization ID associated with your RedHat account with RHEL license for Virtual Datacenters.
`BareMetalHost`	For the bare metal provider only. Information about hardware configuration of a machine. Required for further machine selection during bootstrap. For details, see API Reference: BareMetalHost.
`BareMetalHostCredential`	For the bare metal provider only. The object is created for each `BareMetalHost` and contains information about the Baseboard Management Controller (`bmc`) credentials. For details, see API Reference: BareMetalHostCredential.
`BareMetalHostProfile`	For the bare metal provider only. Provisioning and configuration settings of the storage devices and the operating system. For details, see API Reference: BareMetalHostProfile.
`L2Template`	For the bare metal provider only. Advanced host networking configuration for clusters, which enables, for example, creation of bond interfaces on top of physical interfaces on the host or the use of multiple subnets to separate different types of network traffic. For details, see API Reference: L2Template.
`MetalLBConfig`	For the bare metal provider only. Contains a reference to the `MetalLBConfigTemplate` object configuration. The use of both objects together enables a flexible configuration of MetalLB. For details, see API Reference: MetalLBConfig.
`MetalLBConfigTemplate`	For the bare metal provider only. Default object for the MetalLB configuration, which enables the use of `Subnet` objects to define MetalLB IP address pools. For details, see API Reference: MetalLBConfigTemplate.
`Subnet`	For the bare metal provider only. Configuration for IP address allocation for cluster nodes. For details, see API Reference: Subnet.

Deploy a management cluster using CLI¶

The following procedure describes how to prepare and deploy a management cluster using Bootstrap v2 by operating YAML templates available in the kaas-bootstrap/templates/ folder.

To deploy a management cluster using CLI:

Set up a bootstrap cluster.
Export kubeconfig of the kind cluster:
```
export KUBECONFIG=<pathToKindKubeconfig>
```
By default, <pathToKindKubeconfig> is $HOME/.kube/kind-config-clusterapi.
For the bare metal provider, configure BIOS on a bare metal host.
Depending on the selected provider, navigate to one of the following locations:
- Bare metal: kaas-bootstrap/templates/bm
- OpenStack: kaas-bootstrap/templates
- vSphere: kaas-bootstrap/templates/vsphere

Create the BootstrapRegion object by modifying bootstrapregion.yaml.template.

For the OpenStack and vSphere providers only. Create the Credentials object by modifying <providerName>-config.yaml.template.
Configuration of <providerName>-config.yaml.template
1. Set the region name that must match the BootstrapRegion object name.
2. Set the kaas.mirantis.com/regional-credential label to "true" to use the credentials for the management cluster deployment.
  cat vsphere-config.yaml.template --- apiVersion: kaas.mirantis.com/v1alpha1 kind: VsphereCredential metadata: name: cloud-config labels: kaas.mirantis.com/region: <regionName> kaas.mirantis.com/regional-credential: "true" spec: ...
3. Verify that credentials are valid:
  ./kaas-bootstrap/bin/kubectl get vspherecredentials <credsName> \ -o yaml -o jsonpath='{.status.valid}'
  The output of the command must be "true". Otherwise, fix the issue with credentials before proceeding to the next step.
Create the ServiceUser object by modifying serviceusers.yaml.template.
Configuration of serviceusers.yaml.template
Service user is the initial user to create in Keycloak for access to a newly deployed management cluster. By default, it has the global-admin, operator (namespaced), and bm-pool-operator (namespaced) roles.

You can delete serviceuser after setting up other required users with specific roles or after any integration with an external identity provider, such as LDAP.
apiVersion: kaas.mirantis.com/v1alpha1 kind: ServiceUserList items: - apiVersion: kaas.mirantis.com/v1alpha1 kind: ServiceUser metadata: name: <serviceUserUsername> spec: password: value: <serviceUserPassword>

Optional. Prepare any number of additional SSH keys using the following example:

apiVersion: kaas.mirantis.com/v1alpha1
kind: PublicKey
metadata:
  name: <SSHKeyName>
  namespace: default
spec:
  publicKey: |
    <insert your public key here>

Optional. Add the Proxy object using the example below:

apiVersion: kaas.mirantis.com/v1alpha1
kind: Proxy
metadata:
  labels:
    kaas.mirantis.com/region: <regionName>
  name: <proxyName>
  namespace: default
spec:
  ...

The region label must match the BootstrapRegion object name.

Configure and apply the cluster configuration using cluster deployment templates:

In cluster.yaml.template, set mandatory cluster labels:

labels:
  kaas.mirantis.com/provider: <providerName>
  kaas.mirantis.com/region: <regionName>

Configure provider-specific settings as required.

Bare metal

Inspect the default bare metal host profile definition in templates/bm/baremetalhostprofiles.yaml.template and adjust it to fit your hardware configuration. For details, see Customize the default bare metal host profile.
Warning

All data will be wiped during cluster deployment on devices defined directly or indirectly in the fileSystems list of BareMetalHostProfile. For example:
- A raw device partition with a file system on it
- A device partition in a volume group with a logical volume that has a file system on it
- An mdadm RAID device with a file system on it
- An LVM RAID device with a file system on it
The wipe field is always considered true for these devices. The false value is ignored.

Therefore, to prevent data loss, move the necessary data from these file systems to another server beforehand, if required.

In templates/bm/baremetalhosts.yaml.template, update the bare metal host definitions according to your environment configuration. Use the table below for reference.

Manually set all parameters that start with SET_.
Set the name of the bootstrapRegion object from bootstrapregion.yaml.template for the kaas.mirantis.com/region label across all objects listed in templates/bm/baremetalhosts.yaml.template.

Bare metal hosts template mandatory parameters¶
Parameter	Description	Example value
`SET_MACHINE_0_IPMI_USERNAME`	The IPMI user name to access the BMC. 0	`user`
`SET_MACHINE_0_IPMI_PASSWORD`	The IPMI password to access the BMC. 0	`password`
`SET_MACHINE_0_MAC`	The MAC address of the first master node in the PXE network.	`ac:1f:6b:02:84:71`
`SET_MACHINE_0_BMC_ADDRESS`	The IP address of the BMC endpoint for the first master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.	`192.168.100.11`
`SET_MACHINE_1_IPMI_USERNAME`	The IPMI user name to access the BMC. 0	`user`
`SET_MACHINE_1_IPMI_PASSWORD`	The IPMI password to access the BMC. 0	`password`
`SET_MACHINE_1_MAC`	The MAC address of the second master node in the PXE network.	`ac:1f:6b:02:84:72`
`SET_MACHINE_1_BMC_ADDRESS`	The IP address of the BMC endpoint for the second master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.	`192.168.100.12`
`SET_MACHINE_2_IPMI_USERNAME`	The IPMI user name to access the BMC. 0	`user`
`SET_MACHINE_2_IPMI_PASSWORD`	The IPMI password to access the BMC. 0	`password`
`SET_MACHINE_2_MAC`	The MAC address of the third master node in the PXE network.	`ac:1f:6b:02:84:73`
`SET_MACHINE_2_BMC_ADDRESS`	The IP address of the BMC endpoint for the third master node in the cluster. Must be an address from the OOB network that is accessible through the management network gateway.	`192.168.100.13`

0(1,2,3,4,5,6): The parameter requires a user name and password in plain text.

Configure cluster network:

Important

Bootstrap V2 supports only separated PXE and LCM networks.

To ensure successful bootstrap, enable asymmetric routing on the interfaces of the management cluster nodes. This is required because the seed node relies on one network by default, which can potentially cause traffic asymmetry.

In the kernelParameters section of bm/baremetalhostprofiles.yaml.template, set rp_filter to 2. This enables loose mode as defined in RFC3704.
Example configuration of asymmetric routing
... kernelParameters: ... sysctl: # Enables the "Loose mode" for the "k8s-lcm" interface (management network) net.ipv4.conf.k8s-lcm.rp_filter: "2" # Enables the "Loose mode" for the "bond0" interface (PXE network) net.ipv4.conf.bond0.rp_filter: "2" ...
Note

More complicated solutions that are not described in this manual include getting rid of traffic asymmetry, for example:
- Configure source routing on management cluster nodes.
- Plug the seed node into the same networks as the management cluster nodes, which requires custom configuration of the seed node.
Update the network objects definition in templates/bm/ipam-objects.yaml.template according to the environment configuration. By default, this template implies the use of separate PXE and life-cycle management (LCM) networks.
Manually set all parameters that start with SET_.

Example of the default L2 template snippet for a management cluster:

bonds:
  bond0:
    interfaces:
      - {{ nic 0 }}
      - {{ nic 1 }}
    parameters:
      mode: active-backup
      primary: {{ nic 0 }}
    dhcp4: false
    dhcp6: false
    addresses:
      - {{ ip "bond0:mgmt-pxe" }}
vlans:
  k8s-lcm:
    id: SET_VLAN_ID
    link: bond0
    addresses:
      - {{ ip "k8s-lcm:kaas-mgmt" }}
    nameservers:
      addresses: {{ nameservers_from_subnet "kaas-mgmt" }}
    routes:
      - to: 0.0.0.0/0
        via: {{ gateway_from_subnet "kaas-mgmt" }}

In this example, the following configuration applies:

A bond of two NIC interfaces
A static address in the PXE network set on the bond
An isolated L2 segment for the LCM network is configured using the k8s-lcm VLAN with the static address in the LCM network
The default gateway address is in the LCM network

For general concepts of configuring separate PXE and LCM networks for a management cluster, see Separate PXE and management networks. For the latest object templates and variable names to use since Container Cloud 2.24.0, use the following tables.

Network parameters mapping overview¶
Deployment file name	Parameters list to update manually
`ipam-objects.yaml.template`	`SET_LB_HOST` `SET_MGMT_ADDR_RANGE` `SET_MGMT_CIDR` `SET_MGMT_DNS` `SET_MGMT_NW_GW` `SET_MGMT_SVC_POOL` `SET_PXE_ADDR_POOL` `SET_PXE_ADDR_RANGE` `SET_PXE_CIDR` `SET_PXE_SVC_POOL` `SET_VLAN_ID`
`bootstrap.env`	`KAAS_BM_PXE_IP` `KAAS_BM_PXE_MASK` `KAAS_BM_PXE_BRIDGE`

The below table contains examples of mandatory parameter values to set in templates/bm/ipam-objects.yaml.template for the network scheme that has the following networks:

172.16.59.0/24 - PXE network
172.16.61.0/25 - LCM network

Mandatory network parameters of the IPAM objects template¶
Parameter	Description	Example value
`SET_PXE_CIDR`	The IP address of the PXE network in the CIDR notation. The minimum recommended network size is 256 addresses (`/24` prefix length).	`172.16.59.0/24`
`SET_PXE_SVC_POOL`	The IP address range to use for endpoints of load balancers in the PXE network for the Container Cloud services: Ironic-API, DHCP server, HTTP server, and caching server. The minimum required range size is 5 addresses.	`172.16.59.6-172.16.59.15`
`SET_PXE_ADDR_POOL`	The IP address range in the PXE network to use for dynamic address allocation for hosts during inspection and provisioning. The minimum recommended range size is 30 addresses for management cluster nodes if it is located in a separate PXE network segment. Otherwise, it depends on the number of managed cluster nodes to deploy in the same PXE network segment as the management cluster nodes.	`172.16.59.51-172.16.59.200`
`SET_PXE_ADDR_RANGE`	The IP address range in the PXE network to use for static address allocation on each management cluster node. The minimum recommended range size is 6 addresses.	`172.16.59.41-172.16.59.50`
`SET_MGMT_CIDR`	The IP address of the LCM network for the management cluster in the CIDR notation. If managed clusters will have their separate LCM networks, those networks must be routable to the LCM network. The minimum recommended network size is 128 addresses (`/25` prefix length).	`172.16.61.0/25`
`SET_MGMT_NW_GW`	The default gateway address in the LCM network. This gateway must provide access to the OOB network of the Container Cloud cluster and to the Internet to download the Mirantis artifacts.	`172.16.61.1`
`SET_LB_HOST`	The IP address of the externally accessible MKE API endpoint of the cluster in the CIDR notation. This address must be within the management `SET_MGMT_CIDR` network but must NOT overlap with any other addresses or address ranges within this network. External load balancers are not supported.	`172.16.61.5/32`
`SET_MGMT_DNS`	An external (non-Kubernetes) DNS server accessible from the LCM network.	`8.8.8.8`
`SET_MGMT_ADDR_RANGE`	The IP address range that includes addresses to be allocated to bare metal hosts in the LCM network for the management cluster. When this network is shared with managed clusters, the size of this range limits the number of hosts that can be deployed in all clusters sharing this network. When this network is solely used by a management cluster, the range must include at least 6 addresses for bare metal hosts of the management cluster.	`172.16.61.30-172.16.61.40`
`SET_MGMT_SVC_POOL`	The IP address range to use for the externally accessible endpoints of load balancers in the LCM network for the Container Cloud services, such as Keycloak, web UI, and so on. The minimum required range size is 19 addresses.	`172.16.61.10-172.16.61.29`
`SET_VLAN_ID`	The VLAN ID used for isolation of LCM network. The bootstrap.sh process and the seed node must have routable access to the network in this VLAN.	`3975`

When using separate PXE and LCM networks, the management cluster services are exposed in different networks using two separate MetalLB address pools:

Services exposed through the PXE network are as follows:
- Ironic API as a bare metal provisioning server
- HTTP server that provides images for network boot and server provisioning
- Caching server for accessing the Container Cloud artifacts deployed on hosts
Services exposed through the LCM network are all other Container Cloud services, such as Keycloak, web UI, and so on.

The default MetalLB configuration described in the MetalLBConfigTemplate object template of templates/bm/ipam-objects.yaml.template uses two separate MetalLB address pools. Also, it uses the interfaces selector in its l2Advertisements template.

Caution

When you change the L2Template object template in templates/bm/ipam-objects.yaml.template, ensure that interfaces listed in the interfaces field of the MetalLBConfigTemplate.spec.templates.l2Advertisements section match those used in your L2Template. For details about the interfaces selector, see API Reference: MetalLBConfigTemplate spec.

See Configure MetalLB for details on MetalLB configuration.

In cluster.yaml.template, update the cluster-related settings to fit your deployment.
Optional. Enable WireGuard for traffic encryption on the Kubernetes workloads network.
WireGuard configuration
1. Ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.
2. In templates/bm/cluster.yaml.template, enable WireGuard by adding the secureOverlay parameter:
  spec: ... providerSpec: value: ... secureOverlay: true
  Caution
  
  Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.
For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

vSphere

Modify templates/vsphere/vsphere-config.yaml.template:

Note

Contact your vSphere administrator to provide you with the values for the below parameters.

vSphere configuration data¶
Parameter	Description
`SET_VSPHERE_SERVER`	IP address or FQDN of the vCenter Server.
`SET_VSPHERE_SERVER_PORT`	Port of the vCenter Server. For example, `port: "8443"`. Leave empty to use `"443"` by default.
`SET_VSPHERE_DATACENTER`	vSphere data center name.
`SET_VSPHERE_SERVER_INSECURE`	Flag that controls validation of the vSphere Server certificate. Must be `true` or `false`.
`SET_VSPHERE_CAPI_PROVIDER_USERNAME`	vSphere Cluster API provider user name that you added when preparing the deployment user setup and permissions.
`SET_VSPHERE_CAPI_PROVIDER_PASSWORD`	vSphere Cluster API provider user password.
`SET_VSPHERE_CLOUD_PROVIDER_USERNAME`	vSphere Cloud Provider deployment user name that you added when preparing the deployment user setup and permissions.
`SET_VSPHERE_CLOUD_PROVIDER_PASSWORD`	vSphere Cloud Provider deployment user password.

Modify templates/vsphere/cluster.yaml.template:

vSphere cluster network parameters

Modify the following required network parameters:

Required parameters¶
Parameter	Description
`SET_LB_HOST`	IP address from the provided vSphere network for Kubernetes API load balancer (Keepalived VIP).
`SET_VSPHERE_DATASTORE`	Name of the vSphere datastore. You can use different datastores for vSphere Cluster API and vSphere Cloud Provider.
`SET_VSPHERE_MACHINES_FOLDER`	Path to a folder where the cluster machines metadata will be stored.
`SET_VSPHERE_NETWORK_PATH`	Path to a network for cluster machines.
`SET_VSPHERE_RESOURCE_POOL_PATH`	Path to a resource pool in which VMs will be created.

Note

To obtain the LB_HOST parameter for the selected vSphere network, contact your vSphere administrator who provides you with the IP ranges dedicated to your environment.

Modify other parameters if required. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

For either DHCP or non-DHCP vSphere network:

Determine the vSphere network parameters as described in VMware vSphere network objects and IPAM recommendations.

Provide the following additional parameters for a proper network setup on machines using embedded IP address management (IPAM) in templates/vsphere/cluster.yaml.template:

Note

To obtain IPAM parameters for the selected vSphere network, contact your vSphere administrator who provides you with IP ranges dedicated to your environment only.

vSphere configuration data¶
Parameter	Description
`ipamEnabled`	Enables IPAM. Recommended value is `true` for either DHCP or non-DHCP networks.
`SET_VSPHERE_NETWORK_CIDR`	CIDR of the provided vSphere network. For example, `10.20.0.0/16`.
`SET_VSPHERE_NETWORK_GATEWAY`	Gateway of the provided vSphere network.
`SET_VSPHERE_CIDR_INCLUDE_RANGES`	IP range for the cluster machines. Specify the range of the provided CIDR. For example, `10.20.0.100-10.20.0.200`. If the DHCP network is used, this range must not intersect with the DHCP range of the network.
`SET_VSPHERE_CIDR_EXCLUDE_RANGES`	Optional. IP ranges to be excluded from being assigned to the cluster machines. The MetalLB range and `SET_LB_HOST` should not intersect with the addresses for IPAM. For example, `10.20.0.150-10.20.0.170`.
`SET_VSPHERE_NETWORK_NAMESERVERS`	List of nameservers for the provided vSphere network.

Configure MetalLB parameters:
1. Open the required configuration file for editing:
  
  Since Container Cloud 2.24.0
  
  Open templates/vsphere/metallbconfig.yaml.template. For a detailed MetalLBConfig object description, see API Reference: MetalLBConfig resource.
  
  Before Container Cloud 2.24.0
  
  Open templates/vsphere/cluster.yaml.template.
2. Add SET_VSPHERE_METALLB_RANGE that is the MetalLB range of IP addresses to assign to load balancers for Kubernetes Services.
  
  Note
  
  To obtain the VSPHERE_METALLB_RANGE parameter for the selected vSphere network, contact your vSphere administrator who provides you with the IP ranges dedicated to your environment.
For RHEL deployments, fill out templates/vsphere/rhellicenses.yaml.template.
RHEL license configuration
Use one of the following set of parameters for RHEL machines subscription:
- The user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters.
  
  Optionally, provide the subscription allocation pools to use for the RHEL subscription activation. If not needed, remove the poolIDs field for subscription-manager to automatically select the licenses for machines.
  
  For example:
  spec: username: <username> password: value: <password> poolIDs: - <pool1> - <pool2>
- The activation key and organization ID associated with your RedHat account with RHEL license for Virtual Datacenters. The activation key can be created by the organization administrator on the RedHat Customer Portal.
  
  If you use the RedHat Satellite server for management of your RHEL infrastructure, you can provide a pre-generated activation key from that server. In this case:
  - Provide the URL to the RedHat Satellite RPM for installation of the CA certificate that belongs to that server.
  - Configure squid-proxy on the management or regional cluster to allow access to your Satellite server. For details, see Configure squid-proxy.
  For example:
  spec: activationKey: value: <activation key> orgID: "<organization ID>" rpmUrl: <rpm url>
  Caution
  
  For RHEL 8.7, verify mirrors configuration for your activation key. For more details, see RHEL 8 mirrors configuration.
Warning

Provide only one set of parameters. Mixing the parameters from different activation methods will cause deployment failure.
For CentOS deployments, in templates/vsphere/rhellicenses.yaml.template, remove all lines under items:.
In templates/vsphere/vspherevmtemplate.yaml.template, set the following mandatory parameters:
```
spec:
  packerImageOSName: SET_OS_NAME
  packerImageOSVersion: SET_OS_VERSION
  packerISOImage: SET_ISO_IMAGE
  vsphereCredentialsName: default/cloud-config
  vsphereClusterName: SET_VSPHERE_CLUSTER_NAME
  vsphereNetwork: SET_VSPHERE_NETWORK_PATH
  vsphereDatastore: SET_VSPHERE_DATASTORE_PATH
  vsphereFolder: SET_VSPHERE_FOLDER_PATH
  vsphereResourcePool: SET_VSPHERE_RESOURCE_POOL_PATH
```
For the parameters description, refer to VsphereVMTemplate configuration. You can also configure optional parameters if required.

Caution

For the vsphereCredentialsName and proxyName fields, use names of the corresponding objects previously created using this procedure.

For the rhelLicenseName field, make sure to create the corresponding RHEL license before proceeding to the next step.

Apply the configured template:
```
kubectl apply -f templates/vsphere/vspherevmtemplate.yaml.template
```

Configure StackLight. For parameters description, see StackLight configuration parameters.
Optional. Configure additional cluster settings as described in Configure optional cluster settings.

Apply configuration for machines using machines.yaml.template.
Configuration of machines.yaml.template
1. Add the following mandatory machine labels:
  labels: kaas.mirantis.com/provider: <providerName> cluster.sigs.k8s.io/cluster-name: <clusterName> kaas.mirantis.com/region: <regionName> cluster.sigs.k8s.io/control-plane: "true"
2. Configure the provider-specific settings:
  
  Bare metal
  
  Inspect the machines.yaml.template and adjust spec and labels of each entry according to your deployment. Adjust spec.providerSpec.value.hostSelector values to match BareMetalHost corresponding to each machine. For details, see API Reference: Bare metal Machine spec.
  OpenStack
  
  In templates/machines.yaml.template, modify the spec:providerSpec:value section for 3 control plane nodes marked with the cluster.sigs.k8s.io/control-plane label by substituting the flavor and image parameters with the corresponding values of the control plane nodes in the related OpenStack cluster. For example:
  
  spec: &cp_spec providerSpec: value: apiVersion: "openstackproviderconfig.k8s.io/v1alpha1" kind: "OpenstackMachineProviderSpec" flavor: kaas.minimal image: bionic-server-cloudimg-amd64-20190612
  
  Note
  
  The flavor parameter value provided in the example above is cloud-specific and must meet the Container Cloud requirements.
  
  Also, modify other parameters as required.
  vSphere
  
  In templates/vsphere/machines.yaml.template, define the following parameters:
  
  rhelLicense
  RHEL license name defined in rhellicenses.yaml.template, defaults to kaas-mgmt-rhel-license. Remove or comment out this parameter for CentOS and Ubuntu deployments.
  
  diskGiB
  Disk size in GiB for machines that must match the disk size of the VM template. You can leave this parameter commented to use the disk size of the VM template. The minimum requirement is 120 GiB.
  
  template
  Path to the VM template prepared in the previous step.
  
  Sample template:
  
  spec: providerSpec: value: apiVersion: vsphere.cluster.k8s.io/v1alpha1 kind: VsphereMachineProviderSpec rhelLicense: <rhelLicenseName> numCPUs: 8 memoryMiB: 32768 # diskGiB: 120 template: <vSphereVMTemplatePath>
  
  Also, modify other parameters if required.
For the bare metal provider, monitor the inspecting process of the baremetal hosts and wait until all hosts are in the available state:
```
kubectl get bmh -o go-template='{{- range .items -}} {{.status.provisioning.state}}{{"\n"}} {{- end -}}'
```
Example of system response:
```
available
available
available
```
Monitor the BootstrapRegion object status and wait until it is ready.
```
kubectl get bootstrapregions <regionName> -o go-template='{{.status.ready}}{{"\n"}}'
```
To obtain more granular status details, monitor status.conditions:
```
kubectl get bootstrapregions <regionName> -o jsonpath='{.status.conditions}{"\n"}'
```
For a more convenient system response, consider using dedicated tools such as jq or yq and adjust the -o flag to output in json or yaml format accordingly.

Note

For the bare metal provider, the BareMetalObjectReferences condition is not mandatory and may remain in the not ready state with no effect on the BootstrapRegion object.
Approve the BootstrapRegion object to start the cluster deployment:
```
./kaas-bootstrap/container-cloud bootstrap approve <bootstrapRegionName>
```
Caution

Once you approve the BootstrapRegion object, no cluster or machine modification is allowed.
Monitor the deployment progress. For deployment stages description, see Overview of the deployment workflow.
Verify that network addresses used on your clusters do not overlap with the following default MKE network addresses for Swarm and MCR:
- 10.0.0.0/16 is used for Swarm networks. IP addresses from this network are virtual.
- 10.99.0.0/16 is used for MCR networks. IP addresses from this network are allocated on hosts.
Verification of Swarm and MCR network addresses
To verify Swarm and MCR network addresses, run on any master node:
docker info
Example of system response:
Server: ... Swarm: ... Default Address Pool: 10.0.0.0/16 SubnetSize: 24 ... Default Address Pools: Base: 10.99.0.0/16, Size: 20 ...
Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress network is created by default and occupies the 10.0.0.0/24 address block. Also, three MCR networks are created by default and occupy three address blocks: 10.99.0.0/20, 10.99.16.0/20, 10.99.32.0/20.

To verify the actual networks state and addresses in use, run:
docker network ls docker network inspect <networkName>
Optional for the bare metal provider. If you plan to use multiple L2 segments for provisioning of managed cluster nodes, consider the requirements specified in Configure multiple DHCP ranges using Subnet resources.