Bootstrap a management cluster

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your VMware vSphere-based Mirantis Container Cloud management cluster.

To bootstrap a vSphere-based management cluster:

  1. Log in to the bootstrap node running Ubuntu 20.04 that is configured as described in Prerequisites.

  2. Prepare the bootstrap script:

    1. Download and run the Container Cloud bootstrap script:

      apt install wget
wget https://binary.mirantis.com/releases/get_container_cloud.sh
chmod 0755 get_container_cloud.sh
./get_container_cloud.sh

    2. Change the directory to the kaas-bootstrap folder created by the script.

  3. Obtain your license file that will be required during the bootstrap:

    1. Create a user account at www.mirantis.com.

    2. Log in to your account and download the mirantis.lic license file.

    3. Save the license file as mirantis.lic under the kaas-bootstrap directory on the bootstrap node.

    4. Verify that mirantis.lic contains the exact Container Cloud license previously downloaded from www.mirantis.com by decoding the license JWT token, for example, using jwt.io.

      Example of a valid decoded Container Cloud license data with the mandatory license field:

      {
    "exp": 1652304773,
    "iat": 1636669973,
    "sub": "demo",
    "license": {
        "dev": false,
        "limits": {
            "clusters": 10,
            "workers_per_cluster": 10
        },
        "openstack": null
    }
}

      Warning

      The MKE license does not apply to mirantis.lic. For details about MKE license, see MKE documentation.

  4. Prepare deployment templates:

    1. Modify templates/vsphere/vsphere-config.yaml.template:

      vSphere configuration data

      Parameter

      Description

      SET_VSPHERE_SERVER

      IP address or FQDN of the vCenter Server.

      SET_VSPHERE_SERVER_PORT

      Port of the vCenter Server. For example, port: "8443". Leave empty to use 443 by default.

      SET_VSPHERE_DATACENTER

      vSphere data center name.

      SET_VSPHERE_SERVER_INSECURE

      Flag that controls validation of the vSphere Server certificate. Must be true or false.

      SET_VSPHERE_CAPI_PROVIDER_USERNAME

      vSphere Cluster API provider user name that you added when preparing the deployment user setup and permissions.

      SET_VSPHERE_CAPI_PROVIDER_PASSWORD

      vSphere Cluster API provider user password.

      SET_VSPHERE_CLOUD_PROVIDER_USERNAME

      vSphere Cloud Provider deployment user name that you added when preparing the deployment user setup and permissions.

      SET_VSPHERE_CLOUD_PROVIDER_PASSWORD

      vSphere Cloud Provider deployment user password.

    2. Modify the templates/vsphere/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

      Required parameters

      Parameter

      Description

      SET_LB_HOST

      IP address from the provided vSphere network for load balancer (Keepalived).

      SET_VSPHERE_METALLB_RANGE

      MetalLB range of IP addresses that can be assigned to load balancers for Kubernetes Services.

      SET_VSPHERE_DATASTORE

      Name of the vSphere datastore. You can use different datastores for vSphere Cluster API and vSphere Cloud Provider.

      SET_VSPHERE_MACHINES_FOLDER

      Path to a folder where the cluster machines metadata will be stored.

      SET_VSPHERE_NETWORK_PATH

      Path to a network for cluster machines.

      SET_VSPHERE_RESOURCE_POOL_PATH

      Path to a resource pool in which VMs will be created.

    3. For either DHCP or non-DHCP vSphere network:

      1. Determine the vSphere network parameters as described in VMware vSphere network objects and IPAM recommendations.

      2. Provide the following additional parameters for a proper network setup on machines using embedded IP address management (IPAM) in templates/vsphere/cluster.yaml.template

      vSphere configuration data

      Parameter

      Description

      ipamEnabled

      Enables IPAM. Recommended value is true for either DHCP or non-DHCP networks.

      SET_VSPHERE_NETWORK_CIDR

      CIDR of the provided vSphere network. For example, 10.20.0.0/16.

      SET_VSPHERE_NETWORK_GATEWAY

      Gateway of the provided vSphere network.

      SET_VSPHERE_CIDR_INCLUDE_RANGES

      IP range for the cluster machines. Specify the range of the provided CIDR. For example, 10.20.0.100-10.20.0.200. If the DHCP network is used, this range must not intersect with the DHCP range of the network.

      SET_VSPHERE_CIDR_EXCLUDE_RANGES

      Optional. IP ranges to be excluded from being assigned to the cluster machines. The MetalLB range and SET_LB_HOST should not intersect with the addresses for IPAM. For example, 10.20.0.150-10.20.0.170.

      SET_VSPHERE_NETWORK_NAMESERVERS

      List of nameservers for the provided vSphere network.

    4. For RHEL deployments, fill out templates/vsphere/rhellicenses.yaml.template using one of the following set of parameters for RHEL machines subscription:

      • The user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters.

        Optionally, provide the subscription allocation pools to use for the RHEL subscriptions activation. If not needed, remove the poolIDs field for subscription-manager to automatically select the licenses for machines.

        For example:

        spec:
  username: <username>
  password:
    value: <password>
  poolIDs:
  - <pool1>
  - <pool2>

      • The activation key and organization ID associated with your RedHat account with RHEL license for Virtual Datacenters. The activation key can be created by the organization administrator on RedHat Customer Portal.

        If you use the RedHat Satellite server for management of your RHEL infrastructure, you can provide a pre-generated activation key from that server. In this case:

        • Provide the URL to the RedHat Satellite RPM for installation of the CA certificate that belongs to that server.

        • Configure squid-proxy on the management or regional cluster to allow access to your Satellite server. For details, see Configure squid-proxy.

        For example:

        spec:
  activationKey:
    value: <activation key>
  orgID: "<organization ID>"
  rpmUrl: <rpm url>

        Caution

        For RHEL 8.4 TechPreview, verify mirrors configuration for your activation key. For more details, see RHEL 8 mirrors configuration.

      Caution

      Provide only one set of parameters. Mixing of parameters from different activation methods will cause deployment failure.

    5. For CentOS deployments, in templates/vsphere/rhellicenses.yaml.template, remove all lines under items:.

  5. In bootstrap.env, add the KAAS_VSPHERE_ENABLED=true environment variable that enables the vSphere provider deployment in Container Cloud.

  6. Optional if servers from the Ubuntu NTP pool (*.ubuntu.pool.ntp.org) are accessible from the node where the management cluster is being provisioned. Otherwise, this step is mandatory.

    Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

    In templates/vsphere/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

    spec:
  ...
  providerSpec:
    value:
      kaas:
      ...
        regional:
          - helmReleases:
            - name: vsphere-provider
              values:
                config:
                  lcm:
                    ...
                    ntp:
                      servers:
                      - 0.pool.ntp.org
                      ...
            provider: vsphere
            ...

  7. Prepare the OVF template as described in Prepare the OVF template.

  8. In templates/vsphere/machines.yaml.template:

    • Define SET_VSPHERE_TEMPLATE_PATH prepared in the previous step

    • Modify other parameters as required

    spec:
  providerSpec:
    value:
      apiVersion: vsphere.cluster.k8s.io/v1alpha1
      kind: VsphereMachineProviderSpec
      rhelLicense: <rhel-license-name>
      network:
        devices:
        - dhcp4: true
          dhcp6: false
      template: <SET_VSPHERE_TEMPLATE_PATH>

    Note

    The <rhel-license-name> value is the RHEL license name defined in rhellicenses.yaml.tempalte, defaults to kaas-mgmt-rhel-license. Remove or comment out this parameter for CentOS deployments.

  9. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the management and regional cluster using proxy:

    • HTTP_PROXY

    • HTTPS_PROXY

    • NO_PROXY

    • PROXY_CA_CERTIFICATE_PATH

    Example snippet:

    export HTTP_PROXY=http://proxy.example.com:3128
export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
export NO_PROXY=172.18.10.0,registry.internal.lan
export PROXY_CA_CERTIFICATE_PATH="/home/ubuntu/.mitmproxy/mitmproxy-ca-cert.cer"

    The following formats of variables are accepted:

    Proxy configuration data

    Variable

    Format
    HTTP_PROXY
    HTTPS_PROXY

    • http://proxy.example.com:port - for anonymous access

    • http://user:password@proxy.example.com:port - for restricted access

    NO_PROXY

    Comma-separated list of IP addresses or domain names

    PROXY_CA_CERTIFICATE_PATH Available since 2.20.0 as GA

    Optional. Path to the proxy CA certificate for man-in-the-middle (MITM) proxies. Must be placed on the bootstrap node to be trusted. For details, see Install a CA certificate for a MITM proxy on a bootstrap node.

    Warning

    If you require Internet access to go through a MITM proxy, ensure that the proxy has streaming enabled as described in Enable streaming for MITM.

    Note

    • Since Container Cloud 2.20.0, this parameter is generally available for the OpenStack, bare metal, Equinix Metal with private networking, AWS, and vSphere providers

    • For MOSK-based deployments, the feature support is available since MOSK 22.4

    • Since Container Cloud 2.18.0, this parameter is available as TechPreview for the OpenStack and bare metal providers only

    • For Azure and Equinix Metal with public networking, the feature is not supported

    For implementation details, see Proxy and cache support.

    Caution

    In MITM proxy deployments, RHEL machines registration using the Red Hat portal fails due to the known issue 26070. Therefore, use the internal Red Hat Satellite server that a VM can access directly without a MITM proxy.

    For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Requirements for a VMware vSphere-based cluster.

  10. Optional. Configure external identity provider for IAM.

  11. Optional, Technology Preview in Container Cloud 2.18.0. Removed in Container Cloud 2.19.0 for compatibility reasons, currently not supported. Enables encryption for the Kubernetes workloads network using the following field to the Cluster object spec:

     spec:
   providerSpec:
     value:
       secureOverlay: true

    For more details, see MKE documentation: Kubernetes network encryption.

    • When the option is enabled, Calico networking is configured to use IP-in-IP overlay and BGP routing.

    • When the option is disabled, Calico networking is configured to use VXLAN overlay (no BGP).

  12. Run the bootstrap script:

    ./bootstrap.sh all

    • In case of deployment issues, refer to Troubleshooting and inspect logs.

    • If the script fails for an unknown reason:

      1. Run the cleanup script:

        ./bootstrap.sh cleanup

      2. Rerun the bootstrap script.

  13. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private ssh_key for access to the management cluster nodes that is located in the same directory as the bootstrap script.

      Note

      If the initial version of your Container Cloud management cluster was earlier than 2.6.0, ssh_key is named openstack_tmp and is located at ~/.ssh/.

    • The URL for the Container Cloud web UI.

      To create users with permissions required for accessing the Container Cloud web UI, see Create initial users after a management cluster bootstrap.

    • The StackLight endpoints. For details, see Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) and communicate with Keycloak to authenticate users. Keycloak is exposed using HTTPS and self-signed TLS certificates that are not trusted by web browsers.

    To use your own TLS certificates for Keycloak, refer to Configure TLS certificates for cluster applications.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create and operate a VMware vSphere-based managed cluster.

See also