Create a management cluster for the vSphere provider¶

Available since 2.25.0

This section describes how to create a vSphere-based management cluster using the Container Cloud Bootstrap web UI.

To create a vSphere-based management cluster:

Set up a bootstrap cluster.
Prepare the VMware deployment user setup and permissions.
Open the Container Cloud Bootstrap web UI.

Create a bootstrap region.

Bootstrap region configuration

In the Bootstrap Regions tab, create a bootstrap region:

Set the region name.
Select the required provider.
Optional. Recommended. Leave the Guided Bootstrap Region configuration check box selected. It enables the cluster creation helper in the next window with a series of guided steps for a complete setup of a functional management cluster.

The cluster creation helper contains the same configuration windows as in separate tabs of the left-side menu, but the helper enables the configuration of essential provider components one-by-one inside one modal window.

If you select this option, use the corresponding steps of this procedure described below for description of each tab in Guided Bootstrap Region configuration.

Caution

If no VM templates are present in the vSphere Datacenter, deselect this check box, because VM template configuration is not currently supported by this helper and will be added in one of the following releases.
Click Save.

In the Status column of the Bootstrap regions page, monitor the bootstrap region readiness by hovering over the status icon of the bootstrap region.

Once the orange blinking status icon becomes green and Ready, the bootstrap region deployment is complete. If the cluster status is Error, refer to Troubleshooting.

You can monitor live deployment status of the following bootstrap region components:

Component	Status description
Helm	Installation status of bootstrap Helm releases
Provider	Status of provider configuration and installation for related charts and Deployments
Deployments	Readiness of all Deployments in the bootstrap cluster

Configure credentials for the new cluster.

Credentials configuration

In the Credentials tab:

Click Add Credential to add your vSphere credentials. You can either upload your vsphere.yaml configuration file or fill in the fields manually:

Credentials parameters¶
Parameter	Description
Name	Credentials name.
Provider	Provider name. Select `vsphere`.
Region	Region name. Select the bootstrap region name.
Insecure	Flag that controls validation of the vSphere Server certificate.
Server	IP address or FQDN of the vCenter Server.
Port	Port of the vCenter Server. For example, `port: "443"`.
Datacenter	vSphere data center name.
Cloud provider username	Deployment user name of the vSphere Cloud Provider that you added when preparing the deployment user setup and permissions.
Cloud provider password	Deployment user password for the vSphere Cloud Provider.
ClusterAPI username	User name of the vSphere Cluster API provider that you added when preparing the deployment user setup and permissions.
ClusterAPI password	User password of the vSphere Cluster API provider.

Click Create.
Verify that the new credentials status is Ready. If the status is Error, hover over the status to determine the reason of the issue.

Optional. In the SSH Keys tab, click Add SSH Key to upload the public SSH key(s) for VMs creation.

Mandatory for RHEL-based deployments.

RHEL License configuration

In the RHEL Licenses tab, click Add RHEL License and fill out the form with the following parameters:

RHEL license parameters¶
Parameter	Description
RHEL License Name	RHEL license name
Username (User/Password Registration)	User name to access the RHEL license
Password (User/Password Registration)	Password to access the RHEL license
Organization ID (Activation Key)	Organization key to register a user by
Activation Key (Activation Key)	Activation key to use for user registration
RPM URL (Activation Key)	Optional. URL from which to download RPM packages using RPM Package Manager
Pool IDs	Optional. Specify the pool IDs for RHEL licenses for Virtual Datacenters. Otherwise, Subscription Manager will select a subscription from the list of available and appropriate for the machines.

Mandatory for offline environments with no direct access to the Internet. Otherwise, optional. Enable proxy access to the cluster. Such configuration usually contains proxy for the bootstrap cluster and already has the bootstrap-proxy object to use in the cluster configuration by default.

Proxy configuration

In the Proxies tab, configure proxy:

Click Add Proxy.

In the Add New Proxy wizard, fill out the form with the following parameters:

Proxy configuration¶
Parameter	Description
Proxy Name	Name of the proxy server to use during cluster creation.
Region	From the drop-down list, select the required region.
HTTP Proxy	Add the HTTP proxy server domain name in the following format: `http://proxy.example.com:port` - for anonymous access `http://user:password@proxy.example.com:port` - for restricted access
HTTPS Proxy	Add the HTTPS proxy server domain name in the same format as for HTTP Proxy.
No Proxy	Comma-separated list of IP addresses or domain names. Mandatory to add `host[:port]` of the vCenter server.

For implementation details, see Proxy and cache support.

If your proxy requires a trusted CA certificate, select the CA Certificate check box and paste a CA certificate for a MITM proxy to the corresponding field or upload a certificate using Upload Certificate.

For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Requirements for a VMware vSphere-based cluster.

In the Clusters tab, click Create Cluster and fill out the form with the following parameters:

Cluster configuration

Add Cluster name.
Set the provider Service User Name and Service User Password.

Service user is the initial user to create in Keycloak for access to a newly deployed management cluster. By default, it has the global-admin, operator (namespaced), and bm-pool-operator (namespaced) roles.

You can delete serviceuser after setting up other required users with specific roles or after any integration with an external identity provider, such as LDAP.

Configure general provider settings and Kubernetes parameters:

Provider and Kubernetes configuration

Section

Parameter

Description

General Settings

Provider

Select vSphere.

Provider Credential

From the drop-down list, select the vSphere credentials name that you have previously created.

Release Version

The Container Cloud version.

Proxy

Optional. From the drop-down list, select the proxy server name that you have previously created.

SSH Keys

From the drop-down list, select the SSH key name(s) that you have previously added for the SSH access to VMs.

Container Registry

From the drop-down list, select the Docker registry name that you have previously added using the Container Registries tab. For details, see Define a custom CA certificate for a private Docker registry.

Kubernetes

Node CIDR

Kubernetes nodes CIDR block. For example, 10.10.10.0/24.

Services CIDR Blocks

Kubernetes Services CIDR block. For example, 10.233.0.0/18.

Pods CIDR Blocks

Kubernetes pods CIDR block. For example, 10.233.64.0/18.

Note

The network subnet size of Kubernetes pods influences the number of nodes that can be deployed in the cluster. The default subnet size /18 is enough to create a cluster with up to 256 nodes. Each node uses the /26 address blocks (64 addresses), at least one address block is allocated per node. These addresses are used by the Kubernetes pods with hostNetwork: false. The cluster size may be limited further when some nodes use more than one address block.

Provider

LB Host IP

IP address of the load balancer endpoint that will be used to access the Kubernetes API of the new cluster.

LB Address Range

MetalLB range of IP addresses that can be assigned to load balancers for Kubernetes Services.

vSphere

Machine Folder Path

Full path to the folder that will store the cluster machines metadata. Use the drop-down list to select the required item.

Note

Every drop-down list item of the vSphere section represents a short name of a particular vSphere resource, without the datacenter path. The Network Path drop-down list items also represent specific network types. Start typing the item name in the drop-down list field to filter the results and select the required item.

Network Path

Full path to a network for cluster machines. Use the drop-down list to select the required item.

Resource Pool Path

Full path to a resource pool where VMs will be created. Use the drop-down list to select the required item.

Datastore For Cluster

Full path to a storage for VMs disks. Use the drop-down list to select the required item.

Datastore For Cloud Provider

Full path to a storage for Kubernetes volumes. Use the drop-down list to select the required item.

SCSI Controller Type

SCSI controller type for VMs. Leave pvscsi as default.

Enable IPAM

Enables IPAM. Set to true if a vSphere network has no DHCP server. Also, provide the following additional parameters for a proper network setup on machines using embedded IP address management (IPAM):

Network CIDR	CIDR of the provided vSphere network. For example, `10.20.0.0/16`.
Network Gateway	Gateway of the provided vSphere network.
DNS Name Servers	List of nameservers for the provided vSphere network.
Include Ranges	IP range for the cluster machines. Specify the range of the provided CIDR. For example, `10.20.0.100-10.20.0.200`.
Exclude Ranges	Optional. IP ranges to be excluded from being assigned to the cluster machines. The MetalLB range and the load balancer IP address should not intersect with the addresses for IPAM. For example, `10.20.0.150-10.20.0.170`.

Optional General Settings

Enable Secure Overlay

Experimental, not recommended for production deployments. Removed in Cluster releases 16.0.0 and 14.1.0.

Enable WireGuard for traffic encryption on the Kubernetes workloads network.

For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

Parallel Upgrade Of Worker Machines

Available since the Cluster release 14.1.0.

The maximum number of the worker nodes to update simultaneously. It serves as an upper limit on the number of machines that are drained at a given moment of time. Defaults to 1.

You can configure this option after deployment before the cluster update.

Parallel Preparation For Upgrade Of Worker Machines

Available since the Cluster release 14.1.0.

The maximum number of worker nodes being prepared at a given moment of time, which includes downloading of new artifacts. It serves as a limit for the network load that can occur when downloading the files to the nodes. Defaults to 50.

You can configure this option after deployment before the cluster update.

Configure StackLight:

StackLight configuration

Section	Parameter name	Description
StackLight	Enable Monitoring	Enabled by default, cannot be disabled. Enables StackLight monitoring deployment.
	Enable Logging	Enabled by default, cannot be disabled. Enables StackLight logging stack. For details about the logging components, see Deployment architecture. Note The logging mechanism performance depends on the cluster log load. In case of a high load, you may need to increase the default resource requests and limits for `fluentdLogs`. For details, see StackLight configuration parameters: Resource limits.
	HA Mode	Enabled by default, cannot be disabled. Enables StackLight monitoring in the HA mode. For the differences between HA and non-HA modes, see Deployment architecture.
	StackLight Default Logs Severity Level	Log severity (verbosity) level for all StackLight components. The default value for this parameter is Default component log level that respects original defaults of each StackLight component. For details about severity levels, see Log verbosity.
	StackLight Component Logs Severity Level	The severity level of logs for a specific StackLight component that overrides the value of the StackLight Default Logs Severity Level parameter. For details about severity levels, see Log verbosity. Expand the drop-down menu for a specific component to display its list of available log levels.
OpenSearch	Logstash Retention Time	Available if you select Enable Logging. Specifies the `logstash-*` index retention time.
	Events Retention Time	Available if you select Enable Logging. Specifies the `kubernetes_events-*` index retention time.
	Notifications Retention	Available if you select Enable Logging. Specifies the `notification-*` index retention time and is used for Mirantis OpenStack for Kubernetes.
	Persistent Volume Claim Size	Available if you select Enable Logging. The OpenSearch persistent volume claim size.
	Collected Logs Severity Level	Available if you select Enable Logging. The minimum severity of all Container Cloud components logs collected in OpenSearch. For details about severity levels, see Logging.
Prometheus	Retention Time	The Prometheus database retention period.
	Retention Size	The Prometheus database retention size.
	Persistent Volume Claim Size	The Prometheus persistent volume claim size.
	Enable Watchdog Alert	Select to enable the Watchdog alert that fires as long as the entire alerting pipeline is functional.
	Custom Alerts	Specify alerting rules for new custom alerts or upload a YAML file in the following exemplary format: - alert: HighErrorRate expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5 for: 10m labels: severity: page annotations: summary: High request latency For details, see Official Prometheus documentation: Alerting rules. For the list of the predefined StackLight alerts, see Operations Guide: Available StackLight alerts.
StackLight Email Alerts	Enable Email Alerts	Select to enable the StackLight email alerts.
	Send Resolved	Select to enable notifications about resolved StackLight alerts.
	Require TLS	Select to enable transmitting emails through TLS.
	Email alerts configuration for StackLight	Fill out the following email alerts parameters as required: To - the email address to send notifications to. From - the sender address. SmartHost - the SMTP host through which the emails are sent. Authentication username - the SMTP user name. Authentication password - the SMTP password. Authentication identity - the SMTP identity. Authentication secret - the SMTP secret.
StackLight Slack Alerts	Enable Slack alerts	Select to enable the StackLight Slack alerts.
	Send Resolved	Select to enable notifications about resolved StackLight alerts.
	Slack alerts configuration for StackLight	Fill out the following Slack alerts parameters as required: API URL - The Slack webhook URL. Channel - The channel to send notifications to, for example, #channel-for-alerts.
StackLight optional settings	Enable Reference Application	Available since Container Cloud 2.22.0. Enables Reference Application that is a small microservice application that enables workload monitoring on non-MOSK managed clusters.

Click Create.

Configure the VM template:

VM template configuration

In the Clusters tab, click the required cluster name. The cluster page with VM Templates list opens.
Click Create VM Template.

Configure the VM template:

Section	Parameter	Description
General Settings	Name	VM template name.
	OS Name	Operating system name for the VM template: Ubuntu, RHEL, CentOS. Note For RHEL, a RHEL license is required.
	OS Version	Operating system version for the VM template. For the list of supported operating systems and their versions, refer to Requirements for a VMware vSphere-based cluster.
	Region	Previously configured region name. For example, region-one.
	Credentials	Name of previously configured credentials of the Container Cloud cluster.
	Cluster	From the drop-down list, select the name of the related vSphere cluster in vCenter. Caution Do not confuse with the name of the vSphere cluster in Container Cloud.
	Resource pool	Path to the vSphere resource pool.
	Datastore	Datastore to use for the template.
	ISO File Path	Path to the ISO file containing an installation image to clone within a datastore.
	Network	Name of the vSphere network.
	Folder	Path to store the VM template.
Hardware (optional)	CPUs	CPUs number of the template. Minimum number is 8.
	Disk Size (GiB)	Disk size of the template. An integer value is considered as bytes. The minimum size is 120 Gi. You can use human-readable units. For details, see VsphereVMTemplate.
	Memory (GiB)	RAM size of the template. An integer value is considered as bytes. The minimum size is 16 Gi. For details, see VsphereVMTemplate.
Network (optional)	IPv4 Settings	Select either DHCP or static protocol type. Note For a static protocol type, contact your vSphere administrator to provide you with the required network settings.
RHEL Licensing	RHEL License Name	Mandatory for RHEL-based deployments. Select the license added during the RHEL License configuration. For the `RHELLicense` object description, see Overview of the cluster-related objects in the Container Cloud API/CLI.
	Virt-who	Optional. Select to define the user name and password of the `virt-who` service.
Additional Settings (optional)	Proxy	Name of the previously created `Proxy` object.
	Time Zone	Time zone of a machine in the IANA Timezone Database format. For example, `America/New_York`.

Click Create.

Add machines to the bootstrap cluster:

Machines configuration

In the Clusters tab, click the required cluster name. Click the Machines tab.
Click Create Machine.

Fill out the form with the following parameters:

Container Cloud machine configuration¶
Parameter	Description
Count	Specify the odd number of machines to create. Only Manager machines are allowed for a management cluster. Caution The required minimum number of manager machines is three for HA. A cluster can have more than three manager machines but only an odd number of machines. In an even-sized cluster, an additional machine remains in the `Pending` state until an extra manager machine is added. An even number of manager machines does not provide additional fault tolerance but increases the number of node required for etcd quorum.
Template Path	Path to the prepared VM template. Use the drop-down list to select the required item. Every drop-down list item represents a short name of a VM template, without the datacenter path. Start typing the VM template name in the drop-down list field to filter the results. For the list of supported operating systems, refer to Requirements for a VMware vSphere-based cluster. Caution Deployment of a Container Cloud cluster that is based on different operating systems, such as RHEL and CentOS or CentOS and Ubuntu, is not supported. Note Each template in the drop-down list contains a label. If a VM template was initially created using the built-in Packer mechanism, the Container Cloud version has a green label on the right side of the list. Otherwise, a template is marked with the Unknown label. Mirantis recommends using only green-labeled templates for production deployments.
RHEL License	Applies to RHEL deployments only. From the drop-down list, select the RHEL license that you previously added for the cluster being deployed.
VM Memory Size	VM memory size in GB, defaults to 24 GB.
VM CPU Size	VM CPUs number, defaults to 8.

Click Create.

Optional. Using the Container Cloud CLI, modify the provider-specific and other cluster settings as described in Configure optional cluster settings.
Select from the following options to start cluster deployment:
If you use the Guided Bootstrap Region configuration

Click Deploy.

If you use the left-side web UI menu
Approve the previously created bootstrap region using the Container Cloud CLI:
./kaas-bootstrap/container-cloud bootstrap approve <bootstrapRegionName>
Caution

Once you approve the bootstrap region, no cluster or machine modification is allowed.

Monitor the deployment progress of the cluster and machines.

Monitoring of the cluster readiness

To monitor the cluster readiness, hover over the status icon of a specific cluster in the Status column of the Clusters page.

Once the orange blinking status icon becomes green and Ready, the cluster deployment or update is complete.

You can monitor live deployment status of the following cluster components:

Component	Description
Bastion	For the OpenStack-based management or regional clusters, the Bastion node IP address status that confirms the Bastion node creation
Helm	Installation or upgrade status of all Helm releases
Kubelet	Readiness of the node in a Kubernetes cluster, as reported by kubelet
Kubernetes	Readiness of all requested Kubernetes objects
Nodes	Equality of the requested nodes number in the cluster to the number of nodes having the `Ready` LCM status
OIDC	Readiness of the cluster OIDC configuration
StackLight	Health of all StackLight-related objects in a Kubernetes cluster
Swarm	Readiness of all nodes in a Docker Swarm cluster
LoadBalancer	Readiness of the Kubernetes API load balancer
ProviderInstance	Readiness of all machines in the underlying infrastructure (virtual or bare metal, depending on the provider type)
Graceful Reboot	Readiness of a cluster during a scheduled graceful reboot, available since Container Cloud 2.24.0 for non-MOSK clusters
Infrastructure Status	Available since Container Cloud 2.25.0 for bare metal and OpenStack providers. Readiness of the following cluster components: Bare metal: the `MetalLBConfig` object along with MetalLB and DHCP subnets. OpenStack: cluster network, routers, load balancers, and Bastion along with their ports and floating IPs.

For the history of a cluster deployment or update, refer to Inspect the history of a cluster and machine deployment or update.

Monitoring of machines readiness

To monitor machines readiness, use the status icon of a specific machine on the Clusters page.

Quick status
On the Clusters page, in the Managers column. The green status icon indicates that the machine is Ready, the orange status icon indicates that the machine is Updating.
Detailed status
In the Machines section of a particular cluster page, in the Status column. Hover over a particular machine status icon to verify the deploy or update status of a specific machine component.

You can monitor the status of the following machine components:

Component	Description
Kubelet	Readiness of a node in a Kubernetes cluster.
Swarm	Health and readiness of a node in a Docker Swarm cluster.
LCM	LCM readiness status of a node.
ProviderInstance	Readiness of a node in the underlying infrastructure (virtual or bare metal, depending on the provider type).
Graceful Reboot	Readiness of a machine during a scheduled graceful reboot of a cluster, available since Container Cloud 2.24.0 for non-MOSK clusters.
Infrastructure Status	Available since Container Cloud 2.25.0 for the bare metal provider only. Readiness of the `IPAMHost`, `L2Template`, `BareMetalHost`, and `BareMetalHostProfile` objects associated with the machine.

The machine creation starts with the Provision status. During provisioning, the machine is not expected to be accessible since its infrastructure (VM, network, and so on) is being created.

Other machine statuses are the same as the LCMMachine object states:

Uninitialized - the machine is not yet assigned to an LCMCluster.
Pending - the agent reports a node IP address and host name.
Prepare - the machine executes StateItems that correspond to the prepare phase. This phase usually involves downloading the necessary archives and packages.
Deploy - the machine executes StateItems that correspond to the deploy phase that is becoming a Mirantis Kubernetes Engine (MKE) node.
Ready - the machine is being deployed.
Upgrade - the machine is being upgraded to the new MKE version.
Reconfigure - the machine executes StateItems that correspond to the reconfigure phase. The machine configuration is being updated without affecting workloads running on the machine.

Once the status changes to Ready, the deployment of the cluster components on this machine is complete.

You can also monitor the live machine status using API:

kubectl get machines <machineName> -o wide

Example of system response since Container Cloud 2.23.0:

NAME   READY LCMPHASE  NODENAME              UPGRADEINDEX  REBOOTREQUIRED  WARNINGS
demo-0 true  Ready     kaas-node-c6aa8ad3    1             false

For the history of a machine deployment or update, refer to Inspect the history of a cluster and machine deployment or update.

Alternatively, verify machine statuses from the seed node on which the bootstrap cluster is deployed:

Log in to the seed node.
Export KUBECONFIG to connect to the bootstrap cluster:
```
export KUBECONFIG=~/.kube/kind-config-clusterapi
```
Verify the statuses of available LCMMachine objects:
```
kubectl get lcmmachines -o wide
```
Verify the statuses of available cluster machines:
```
kubectl get machines -o wide
```

Verify that network addresses used on your clusters do not overlap with the following default MKE network addresses for Swarm and MCR:
- 10.0.0.0/16 is used for Swarm networks. IP addresses from this network are virtual.
- 10.99.0.0/16 is used for MCR networks. IP addresses from this network are allocated on hosts.
Verification of Swarm and MCR network addresses
To verify Swarm and MCR network addresses, run on any master node:
docker info
Example of system response:
Server: ... Swarm: ... Default Address Pool: 10.0.0.0/16 SubnetSize: 24 ... Default Address Pools: Base: 10.99.0.0/16, Size: 20 ...
Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress network is created by default and occupies the 10.0.0.0/24 address block. Also, three MCR networks are created by default and occupy three address blocks: 10.99.0.0/20, 10.99.16.0/20, 10.99.32.0/20.

To verify the actual networks state and addresses in use, run:
docker network ls docker network inspect <networkName>