Create a management cluster for the vSphere provider

This section describes how to create a vSphere-based management cluster using the Container Cloud Bootstrap web UI.

To create a vSphere-based management cluster:

  1. Set up a bootstrap cluster.

  2. Prepare the VMware deployment user setup and permissions.

  3. Open the Container Cloud Bootstrap web UI.

  4. Create a bootstrap object.

    Bootstrap object configuration

    In the Bootstrap tab, create a bootstrap object:

    1. Set the bootstrap object name.

    2. Select the required provider.

    3. Optional. Recommended. Leave the Guided Bootstrap configuration check box selected. It enables the cluster creation helper in the next window with a series of guided steps for a complete setup of a functional management cluster.

      The cluster creation helper contains the same configuration windows as in separate tabs of the left-side menu, but the helper enables the configuration of essential provider components one-by-one inside one modal window.

      If you select this option, use the corresponding steps of this procedure described below for description of each tab in Guided Bootstrap configuration.

      Caution

      If no VM templates are present in the vSphere Datacenter, deselect this check box, because VM template configuration is not currently supported by this helper and will be added in one of the following releases.

    4. Click Save.

    5. In the Status column of the Bootstrap page, monitor the bootstrap region readiness by hovering over the status icon of the bootstrap region.

      Once the orange blinking status icon becomes green and Ready, the bootstrap region deployment is complete. If the cluster status is Error, refer to Troubleshooting.

      You can monitor live deployment status of the following bootstrap region components:

      Component

      Status description

      Helm

      Installation status of bootstrap Helm releases

      Provider

      Status of provider configuration and installation for related charts and Deployments

      Deployments

      Readiness of all Deployments in the bootstrap cluster

  5. Configure credentials for the new cluster.

    Credentials configuration

    In the Credentials tab:

    1. Click Add Credential to add your vSphere credentials. You can either upload your vsphere.yaml configuration file or fill in the fields manually:

      Credentials parameters

      Parameter

      Description

      Name

      Credentials name.

      Provider

      Provider name. Select vsphere.

      Region

      Region name. Select the bootstrap region name.

      Insecure

      Flag that controls validation of the vSphere Server certificate.

      Server

      IP address or FQDN of the vCenter Server.

      Port

      Port of the vCenter Server. For example, port: "443".

      Datacenter

      vSphere data center name.

      Cloud provider username

      Deployment user name of the vSphere Cloud Provider that you added when preparing the deployment user setup and permissions.

      Cloud provider password

      Deployment user password for the vSphere Cloud Provider.

      ClusterAPI username

      User name of the vSphere Cluster API provider that you added when preparing the deployment user setup and permissions.

      ClusterAPI password

      User password of the vSphere Cluster API provider.

    2. Click Create.

    3. Verify that the new credentials status is Ready. If the status is Error, hover over the status to determine the reason of the issue.

  6. Optional. In the SSH Keys tab, click Add SSH Key to upload the public SSH key(s) for VMs creation.

  7. Mandatory for RHEL-based deployments.

    RHEL License configuration

    In the RHEL Licenses tab, click Add RHEL License and fill out the form with the following parameters:

    RHEL license parameters

    Parameter

    Description

    RHEL License Name

    RHEL license name

    Username (User/Password Registration)

    User name to access the RHEL license

    Password (User/Password Registration)

    Password to access the RHEL license

    Organization ID (Activation Key)

    Organization key to register a user by

    Activation Key (Activation Key)

    Activation key to use for user registration

    RPM URL (Activation Key)

    Optional. URL from which to download RPM packages using RPM Package Manager

    Pool IDs

    Optional. Specify the pool IDs for RHEL licenses for Virtual Datacenters. Otherwise, Subscription Manager will select a subscription from the list of available and appropriate for the machines.

  8. Mandatory for offline environments with no direct access to the Internet. Otherwise, optional. Enable proxy access to the cluster. Such configuration usually contains proxy for the bootstrap cluster and already has the bootstrap-proxy object to use in the cluster configuration by default.

    Proxy configuration

    In the Proxies tab, configure proxy:

    1. Click Add Proxy.

    2. In the Add New Proxy wizard, fill out the form with the following parameters:

      Proxy configuration

      Parameter

      Description

      Proxy Name

      Name of the proxy server to use during cluster creation.

      Region Removed in 2.26.0 (16.1.0 and 17.1.0)

      From the drop-down list, select the required region.

      HTTP Proxy

      Add the HTTP proxy server domain name in the following format:

      • http://proxy.example.com:port - for anonymous access

      • http://user:password@proxy.example.com:port - for restricted access

      HTTPS Proxy

      Add the HTTPS proxy server domain name in the same format as for HTTP Proxy.

      No Proxy

      Comma-separated list of IP addresses or domain names. Mandatory to add host[:port] of the vCenter server.

      For implementation details, see Proxy and cache support.

    3. If your proxy requires a trusted CA certificate, select the CA Certificate check box and paste a CA certificate for a MITM proxy to the corresponding field or upload a certificate using Upload Certificate.

    For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Requirements for a VMware vSphere-based cluster.

  9. In the Clusters tab, click Create Cluster and fill out the form with the following parameters:

    Cluster configuration

    1. Add Cluster name.

    2. Set the provider Service User Name and Service User Password.

      Service user is the initial user to create in Keycloak for access to a newly deployed management cluster. By default, it has the global-admin, operator (namespaced), and bm-pool-operator (namespaced) roles.

      You can delete serviceuser after setting up other required users with specific roles or after any integration with an external identity provider, such as LDAP.

    3. Configure general provider settings and Kubernetes parameters:

      Provider and Kubernetes configuration

      Section

      Parameter

      Description

      General Settings

      Provider

      Select vSphere.

      Provider Credential

      From the drop-down list, select the vSphere credentials name that you have previously created.

      Release Version

      The Container Cloud version.

      Caution

      Due to the known issue 40747, the Cluster release 16.0.0, which is not supported since Container Cloud 2.25.1 for greenfield deployments, is still available in the drop-down menu for managed clusters.

      Do not select this Cluster release to prevent deployment failures. Select the latest supported version instead.

      Proxy

      Optional. From the drop-down list, select the proxy server name that you have previously created.

      SSH Keys

      From the drop-down list, select the SSH key name(s) that you have previously added for the SSH access to VMs.

      Container Registry

      From the drop-down list, select the Docker registry name that you have previously added using the Container Registries tab. For details, see Define a custom CA certificate for a private Docker registry.

      Kubernetes

      Node CIDR

      Kubernetes nodes CIDR block. For example, 10.10.10.0/24.

      Services CIDR Blocks

      Kubernetes Services CIDR block. For example, 10.233.0.0/18.

      Pods CIDR Blocks

      Kubernetes pods CIDR block. For example, 10.233.64.0/18.

      Note

      The network subnet size of Kubernetes pods influences the number of nodes that can be deployed in the cluster. The default subnet size /18 is enough to create a cluster with up to 256 nodes. Each node uses the /26 address blocks (64 addresses), at least one address block is allocated per node. These addresses are used by the Kubernetes pods with hostNetwork: false. The cluster size may be limited further when some nodes use more than one address block.

      Provider

      LB Host IP

      IP address of the load balancer endpoint that will be used to access the Kubernetes API of the new cluster.

      LB Address Range

      MetalLB range of IP addresses that can be assigned to load balancers for Kubernetes Services.

      vSphere

      Machine Folder Path

      Full path to the folder that will store the cluster machines metadata. Use the drop-down list to select the required item.

      Note

      Every drop-down list item of the vSphere section represents a short name of a particular vSphere resource, without the datacenter path. The Network Path drop-down list items also represent specific network types. Start typing the item name in the drop-down list field to filter the results and select the required item.

      Network Path

      Full path to a network for cluster machines. Use the drop-down list to select the required item.

      Resource Pool Path

      Full path to a resource pool where VMs will be created. Use the drop-down list to select the required item.

      Datastore For Cluster

      Full path to a storage for VMs disks. Use the drop-down list to select the required item.

      Datastore For Cloud Provider

      Full path to a storage for Kubernetes volumes. Use the drop-down list to select the required item.

      SCSI Controller Type

      SCSI controller type for VMs. Leave pvscsi as default.

      Enable IPAM

      Enables IPAM. Set to true if a vSphere network has no DHCP server. Also, provide the following additional parameters for a proper network setup on machines using embedded IP address management (IPAM):

      Network CIDR

      CIDR of the provided vSphere network. For example, 10.20.0.0/16.

      Network Gateway

      Gateway of the provided vSphere network.

      DNS Name Servers

      List of nameservers for the provided vSphere network.

      Include Ranges

      IP range for the cluster machines. Specify the range of the provided CIDR. For example, 10.20.0.100-10.20.0.200.

      Exclude Ranges

      Optional. IP ranges to be excluded from being assigned to the cluster machines. The MetalLB range and the load balancer IP address should not intersect with the addresses for IPAM. For example, 10.20.0.150-10.20.0.170.

      Optional General Settings

      Enable Secure Overlay

      Experimental, not recommended for production deployments. Removed in Cluster releases 16.0.0 and 14.1.0.

      Enable WireGuard for traffic encryption on the Kubernetes workloads network.

      WireGuard configuration

      1. Ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.

      2. Enable WireGuard by selecting the Enable WireGuard check box.

        Caution

        Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.

      For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

      Parallel Upgrade Of Worker Machines

      Available since the Cluster release 14.1.0.

      The maximum number of the worker nodes to update simultaneously. It serves as an upper limit on the number of machines that are drained at a given moment of time. Defaults to 1.

      You can configure this option after deployment before the cluster update.

      Parallel Preparation For Upgrade Of Worker Machines

      Available since the Cluster release 14.1.0.

      The maximum number of worker nodes being prepared at a given moment of time, which includes downloading of new artifacts. It serves as a limit for the network load that can occur when downloading the files to the nodes. Defaults to 50.

      You can configure this option after deployment before the cluster update.

    4. Configure StackLight:

      StackLight configuration

    5. Click Create.

  10. Configure the VM template:

    VM template configuration

    1. In the Clusters tab, click the required cluster name. The cluster page with VM Templates list opens.

    2. Click Create VM Template.

    3. Configure the VM template:

      Section

      Parameter

      Description

      General Settings

      Name

      VM template name.

      OS Name

      Operating system name for the VM template: Ubuntu or RHEL.

      Note

      For RHEL, a RHEL license is required.

      OS Version

      Operating system version for the VM template. For the list of supported operating systems and their versions, refer to Requirements for a VMware vSphere-based cluster.

      Region

      Previously configured region name. For example, region-one.

      Credentials

      Name of previously configured credentials of the Container Cloud cluster.

      Cluster

      From the drop-down list, select the name of the related vSphere cluster in vCenter.

      Caution

      Do not confuse with the name of the vSphere cluster in Container Cloud.

      Resource pool

      Path to the vSphere resource pool.

      Datastore

      Datastore to use for the template.

      ISO File Path

      Path to the ISO file containing an installation image to clone within a datastore.

      Network

      Name of the vSphere network.

      Folder

      Path to store the VM template.

      Hardware (optional)

      CPUs

      CPUs number of the template. Minimum number is 8.

      Disk Size (GiB)

      Disk size of the template. An integer value is considered as bytes. The minimum size is 120 Gi. You can use human-readable units. For details, see VsphereVMTemplate.

      Memory (GiB)

      RAM size of the template. An integer value is considered as bytes. The minimum size is 16 Gi. For details, see VsphereVMTemplate.

      Network (optional)

      IPv4 Settings

      Select either DHCP or static protocol type.

      Note

      For a static protocol type, contact your vSphere administrator to provide you with the required network settings.

      RHEL Licensing

      RHEL License Name

      Mandatory for RHEL-based deployments. Select the license added during the RHEL License configuration. For the RHELLicense object description, see Overview of the cluster-related objects in the Container Cloud API/CLI.

      Virt-who

      Optional. Select to define the user name and password of the virt-who service.

      Additional Settings (optional)

      Proxy

      Name of the previously created Proxy object.

      Time Zone

      Time zone of a machine in the IANA Timezone Database format. For example, America/New_York.

    4. Click Create.

  11. Add machines to the bootstrap cluster:

    Machines configuration

    1. In the Clusters tab, click the required cluster name. Click the Machines tab.

    2. Click Create Machine.

    3. Fill out the form with the following parameters:

      Container Cloud machine configuration

      Parameter

      Description

      Count

      Specify the odd number of machines to create. Only Manager machines are allowed for a management cluster.

      Caution

      The required minimum number of manager machines is three for HA. A cluster can have more than three manager machines but only an odd number of machines.

      In an even-sized cluster, an additional machine remains in the Pending state until an extra manager machine is added. An even number of manager machines does not provide additional fault tolerance but increases the number of node required for etcd quorum.

      VM Source

      Select Template Object and use the drop-down list to select the VM template name prepared in the previous step.

      If you select vSphere Path, you may also use VM templates of your vSphere datacenter account that are displayed in the drop-down list. For the list of supported operating systems, refer to Requirements for a VMware vSphere-based cluster.

      Note

      Mirantis does not recommend using VM templates that contain the Unknown label in the drop-down list.

      Caution

      Container Cloud does not support mixed operating systems, RHEL combined with Ubuntu, in one cluster.

      RHEL License

      Applies to RHEL deployments only.

      From the drop-down list, select the RHEL license that you previously added for the cluster being deployed.

      VM Memory Size

      VM memory size in GB, defaults to 24 GB.

      VM CPU Size

      VM CPUs number, defaults to 8.

    4. Click Create.

  12. Optional. Using the Container Cloud CLI, modify the provider-specific and other cluster settings as described in Configure optional cluster settings.

  13. Select from the following options to start cluster deployment:

    Click Deploy.

    Approve the previously created bootstrap region using the Container Cloud CLI:

    ./kaas-bootstrap/container-cloud bootstrap approve all

    ./kaas-bootstrap/container-cloud bootstrap approve  <bootstrapRegionName>

    Caution

    Once you approve the bootstrap region, no cluster or machine modification is allowed.

  14. Monitor the deployment progress of the cluster and machines.

    Monitoring of the cluster readiness

    To monitor the cluster readiness, hover over the status icon of a specific cluster in the Status column of the Clusters page.

    Once the orange blinking status icon becomes green and Ready, the cluster deployment or update is complete.

    You can monitor live deployment status of the following cluster components:

    Component

    Description

    Bastion

    For the OpenStack-based management clusters, the Bastion node IP address status that confirms the Bastion node creation

    Helm

    Installation or upgrade status of all Helm releases

    Kubelet

    Readiness of the node in a Kubernetes cluster, as reported by kubelet

    Kubernetes

    Readiness of all requested Kubernetes objects

    Nodes

    Equality of the requested nodes number in the cluster to the number of nodes having the Ready LCM status

    OIDC

    Readiness of the cluster OIDC configuration

    StackLight

    Health of all StackLight-related objects in a Kubernetes cluster

    Swarm

    Readiness of all nodes in a Docker Swarm cluster

    LoadBalancer

    Readiness of the Kubernetes API load balancer

    ProviderInstance

    Readiness of all machines in the underlying infrastructure (virtual or bare metal, depending on the provider type)

    Graceful Reboot

    Readiness of a cluster during a scheduled graceful reboot, available since Cluster releases 15.0.1 and 14.0.0.

    Infrastructure Status

    Available since Container Cloud 2.25.0 for bare metal and OpenStack providers. Readiness of the following cluster components:

    • Bare metal: the MetalLBConfig object along with MetalLB and DHCP subnets.

    • OpenStack: cluster network, routers, load balancers, and Bastion along with their ports and floating IPs.

    LCM Operation

    Available since Container Cloud 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Health of all LCM operations on the cluster and its machines.

    For the history of a cluster deployment or update, refer to Inspect the history of a cluster and machine deployment or update.

    Monitoring of machines readiness

    To monitor machines readiness, use the status icon of a specific machine on the Clusters page.

    • Quick status

      On the Clusters page, in the Managers column. The green status icon indicates that the machine is Ready, the orange status icon indicates that the machine is Updating.

    • Detailed status

      In the Machines section of a particular cluster page, in the Status column. Hover over a particular machine status icon to verify the deploy or update status of a specific machine component.

    You can monitor the status of the following machine components:

    Component

    Description

    Kubelet

    Readiness of a node in a Kubernetes cluster.

    Swarm

    Health and readiness of a node in a Docker Swarm cluster.

    LCM

    LCM readiness status of a node.

    ProviderInstance

    Readiness of a node in the underlying infrastructure (virtual or bare metal, depending on the provider type).

    Graceful Reboot

    Readiness of a machine during a scheduled graceful reboot of a cluster, available since Cluster releases 15.0.1 and 14.0.0.

    Infrastructure Status

    Available since Container Cloud 2.25.0 for the bare metal provider only. Readiness of the IPAMHost, L2Template, BareMetalHost, and BareMetalHostProfile objects associated with the machine.

    LCM Operation

    Available since Container Cloud 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Health of all LCM operations on the machine.

    The machine creation starts with the Provision status. During provisioning, the machine is not expected to be accessible since its infrastructure (VM, network, and so on) is being created.

    Other machine statuses are the same as the LCMMachine object states:

    1. Uninitialized - the machine is not yet assigned to an LCMCluster.

    2. Pending - the agent reports a node IP address and host name.

    3. Prepare - the machine executes StateItems that correspond to the prepare phase. This phase usually involves downloading the necessary archives and packages.

    4. Deploy - the machine executes StateItems that correspond to the deploy phase that is becoming a Mirantis Kubernetes Engine (MKE) node.

    5. Ready - the machine is being deployed.

    6. Upgrade - the machine is being upgraded to the new MKE version.

    7. Reconfigure - the machine executes StateItems that correspond to the reconfigure phase. The machine configuration is being updated without affecting workloads running on the machine.

    Once the status changes to Ready, the deployment of the cluster components on this machine is complete.

    You can also monitor the live machine status using API:

    kubectl get machines <machineName> -o wide

    Example of system response since Container Cloud 2.23.0:

    NAME   READY LCMPHASE  NODENAME              UPGRADEINDEX  REBOOTREQUIRED  WARNINGS
demo-0 true  Ready     kaas-node-c6aa8ad3    1             false

    For the history of a machine deployment or update, refer to Inspect the history of a cluster and machine deployment or update.

    Alternatively, verify machine statuses from the seed node on which the bootstrap cluster is deployed:

    1. Log in to the seed node.

    2. Export KUBECONFIG to connect to the bootstrap cluster:

      export KUBECONFIG=~/.kube/kind-config-clusterapi

    3. Verify the statuses of available LCMMachine objects:

      kubectl get lcmmachines -o wide

    4. Verify the statuses of available cluster machines:

      kubectl get machines -o wide

  15. Verify that network addresses used on your clusters do not overlap with the following default MKE network addresses for Swarm and MCR:

    • 10.0.0.0/16 is used for Swarm networks. IP addresses from this network are virtual.

    • 10.99.0.0/16 is used for MCR networks. IP addresses from this network are allocated on hosts.

    Verification of Swarm and MCR network addresses

    To verify Swarm and MCR network addresses, run on any master node:

    docker info

    Example of system response:

    Server:
 ...
 Swarm:
  ...
  Default Address Pool: 10.0.0.0/16
  SubnetSize: 24
  ...
 Default Address Pools:
   Base: 10.99.0.0/16, Size: 20
 ...

    Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress network is created by default and occupies the 10.0.0.0/24 address block. Also, three MCR networks are created by default and occupy three address blocks: 10.99.0.0/20, 10.99.16.0/20, 10.99.32.0/20.

    To verify the actual networks state and addresses in use, run:

    docker network ls
docker network inspect <networkName>