Mirantis Container Cloud Deployment Guide¶

Workflow overview¶

The bare metal management system enables the Infrastructure Operator to deploy Mirantis Container Cloud on a set of bare metal servers. It also enables Container Cloud to deploy managed clusters on bare metal servers without a pre-provisioned operating system.

The Infrastructure Operator performs the following steps to install Container Cloud in a bare metal environment:

1. Install and connect hardware servers as described in Reference Architecture: Baremetal-based Container Cloud cluster.

   Caution

   The baremetal-based Container Cloud does not manage the underlay networking fabric but requires specific network configuration to operate.

2. Install Ubuntu 18.04 on one of the bare metal machines to create a seed node and copy the bootstrap tarball to this node.

3. Obtain the Mirantis license file that will be required during the bootstrap.

4. Create the deployment configuration files that include the bare metal hosts metadata.

5. Validate the deployment templates using fast preflight.

6. Run the bootstrap script for the fully automated installation of the management cluster onto the selected bare metal hosts.

Using the bootstrap script, the Container Cloud bare metal management system prepares the seed node for the management cluster and starts the deployment of Container Cloud itself. The bootstrap script performs all necessary operations to perform the automated management cluster setup. The deployment diagram below illustrates the bootstrap workflow of a baremetal-based management cluster.

Bootstrap a management cluster¶

This section describes how to prepare and bootstrap a baremetal-based management cluster. The procedure includes:

• A runbook that describes how to create a seed node that is a temporary server used to run the management cluster bootstrap scripts.

• A step-by-step instruction how to prepare metadata for the bootstrap scripts and how to run them.

Customize the default bare metal host profile¶

This section describes the bare metal host profile settings and instructs how to configure this profile before deploying Mirantis Container Cloud on physical servers.

The bare metal host profile is a Kubernetes custom resource. It allows the Infrastructure Operator to define how the storage devices and the operating system are provisioned and configured.

The bootstrap templates for a bare metal deployment include the template for the default BareMetalHostProfile object in the following file that defines the default bare metal host profile:

templates/bm/baremetalhostprofiles.yaml.template

The customization procedure of BareMetalHostProfile is almost the same for the management and managed clusters, with the following differences:

• For a management cluster, the customization automatically applies to machines during bootstrap. And for a managed cluster, you apply the changes using kubectl before creating a managed cluster.

• For a management cluster, you edit the default baremetalhostprofiles.yaml.template. And for a managed cluster, you create a new BareMetalHostProfile with the necessary configuration.

For the procedure details, see Operations Guide: Create a custom bare metal host profile. Use this procedure for both types of clusters considering the differences described above.

Workflow overview¶

The Infrastructure Operator performs the following steps to install Mirantis Container Cloud on an OpenStack-based environment:

1. Prepare an OpenStack environment with the requirements described in Reference Architecture: OpenStack-based cluster requirements.

2. Prepare the bootstrap node using Prerequisites.

3. Obtain the Mirantis license file that will be required during the bootstrap.

4. Prepare the OpenStack clouds.yaml file.

5. Create and configure the deployment configuration files that include the cluster and machines metadata.

6. Run the bootstrap script for the fully automated installation of the management cluster.

For more details, see Bootstrap a management cluster.

Prerequisites¶

Before you start with bootstrapping the OpenStack-based management cluster, complete the following prerequisite steps:

1. Verify that your planned cloud meets the reference hardware bill of material and software requirements as described in Reference Architecture: Requirements for an OpenStack-based Mirantis Container Cloud.

2. Configure the bootstrap node:

   1. Log in to any personal computer or VM running Ubuntu 18.04 that you will be using as the bootstrap node.

   2. If you use a newly created VM, run:

      sudo apt-get update
      

   3. Install the current Docker version available for Ubuntu 18.04:

      sudo apt install docker.io
      

   4. Grant your USER access to the Docker daemon:

      sudo usermod -aG docker $USER
      

   5. Log off and log in again to the bootstrap node to apply the changes.

   6. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://binary.mirantis.com"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

3. Proceed to Bootstrap a management cluster.

Bootstrap a management cluster¶

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your OpenStack-based Mirantis Container Cloud management cluster.

To bootstrap an OpenStack-based management cluster:

1. Log in to the bootstrap node running Ubuntu 18.04 that is configured as described in Prerequisites.

2. Download and run the Container Cloud bootstrap script:

   wget https://binary.mirantis.com/releases/get_container_cloud.sh
   chmod 0755 get_container_cloud.sh
   ./get_container_cloud.sh
   

3. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

4. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

5. Save the license file as mirantis.lic under the kaas-bootstrap directory.

6. Prepare the OpenStack configuration for a new cluster:

   1. Log in to the OpenStack Horizon.

   2. In the Project section, select API Access.

   3. In the right-side drop-down menu Download OpenStack RC File, select OpenStack clouds.yaml File.

   4. Add the downloaded clouds.yaml file to the directory with the bootstrap.sh script.

   5. In clouds.yaml, add the password field with your OpenStack password under the clouds/openstack/auth section.

      Example:

      clouds:
        openstack:
          auth:
            auth_url: https://auth.openstack.example.com:5000/v3
            username: your_username
            password: your_secret_password
            project_id: your_project_id
            user_domain_name: your_user_domain_name
          region_name: RegionOne
          interface: public
          identity_api_version: 3
      

   6. Verify access to the target cloud endpoint from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://auth.openstack.example.com:5000/v3"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

7. Configure the cluster and machines metadata:

   1. Change the directory to the kaas-bootstrap folder.

   2. In templates/machines.yaml.template, modify the spec:providerSpec:value section for 3 control plane nodes marked with the cluster.sigs.k8s.io/control-plane label by substituting the flavor and image parameters with the corresponding values of the control plane nodes in the related OpenStack cluster. For example:

      spec: &cp_spec
        providerSpec:
          value:
            apiVersion: "openstackproviderconfig.k8s.io/v1alpha1"
            kind: "OpenstackMachineProviderSpec"
            flavor: kaas.minimal
            image: bionic-server-cloudimg-amd64-20190612
      

      Also, modify other parameters as required.

   3. Modify the templates/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

   4. ^{Available since 2.5.0} Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

      In templates/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

      spec:
        ...
        providerSpec:
          value:
            kaas:
            ...
              regional:
                - helmReleases:
                  - name: openstack-provider
                    values:
                      config:
                        lcm:
                          ...
                          ntp:
                            servers:
                            - 0.pool.ntp.org
                            ...
      

   Note

   The passwordSalt and passwordHash values for the IAM roles are automatically re-generated during the IAM configuration described below in this procedure.

8. ^{Available since 2.5.0} Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the management and regional cluster using proxy:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

   Example snippet:

   export HTTP_PROXY=http://proxy.example.com:3128
   export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
   export NO_PROXY=172.18.10.0,registry.internal.lan
   

   The following variables formats are accepted:

   vSphere configuration data¶

   Variable	Format
   HTTP_PROXY HTTPS_PROXY	http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
   NO_PROXY	Comma-separated list of IP addresses or domain names

   For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

9. Optional. Skip this step to use the default password password in the Container Cloud web UI.

   Configure the IAM parameters:

   1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

      ./bin/hash-generate -i 27500
      

      The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

      Example of system response:

      passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
      passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
      passwordHashAlgorithm: pbkdf2-sha256
      passwordHashIterations: 27500
      

      Run the tool several times to generate hashed passwords for every IAM role.

   2. Open templates/cluster.yaml.template for editing.

   3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

      • passwordSalt - base64-encoded randomly generated sequence of bytes.

      • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

10. Optional. Configure external identity provider for IAM.

11. Run the bootstrap script:

    ./bootstrap.sh all
    

12. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private SSH key openstack_tmp located in ~/.ssh/ for access to the management cluster nodes.

      Note

      The SSH key name openstack_tmp is the same for all cloud providers. This name will be changed in one of the following Container Cloud releases to avoid confusion with a cloud provider name and its related SSH key name.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The StackLight endpoints. For details, see Operations Guide: Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

13. In case of deployment issues, collect and inspect the bootstrap and management cluster logs as described in Troubleshooting.

14. Optional. Deploy an additional regional cluster as described in Deploy an additional regional cluster (optional).

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create an OpenStack-based managed cluster.

Workflow overview¶

The Infrastructure Operator performs the following steps to install Mirantis Container Cloud on an AWS-based environment:

1. Prepare an AWS environment with the requirements described in Reference Architecture: AWS-based Container Cloud cluster requirements.

2. Prepare the bootstrap node as per Prerequisites.

3. Obtain the Mirantis license file that will be required during the bootstrap.

4. Prepare the AWS environment credentials.

5. Create and configure the deployment configuration files that include the cluster and machines metadata.

6. Run the bootstrap script for the fully automated installation of the management cluster.

For more details, see Bootstrap a management cluster.

Prerequisites¶

Before you start with bootstrapping the AWS-based management cluster, complete the following prerequisite steps:

1. Inspect the Requirements for an AWS-based Container Cloud cluster to understand the potential impact of the Container Cloud deployment on your AWS cloud usage.

2. Configure the bootstrap node:

   1. Log in to any personal computer or VM running Ubuntu 18.04 that you will be using as the bootstrap node.

   2. If you use a newly created VM, run:

      sudo apt-get update
      

   3. Install the current Docker version available for Ubuntu 18.04:

      sudo apt install docker.io
      

   4. Grant your USER access to the Docker daemon:

      sudo usermod -aG docker $USER
      

   5. Log off and log in again to the bootstrap node to apply the changes.

   6. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://binary.mirantis.com"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

3. Proceed to Bootstrap a management cluster.

Bootstrap a management cluster¶

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your AWS-based Mirantis Container Cloud management cluster.

To bootstrap an AWS-based management cluster:

1. Log in to the bootstrap node running Ubuntu 18.04 that is configured as described in Prerequisites.

2. Download and run the Container Cloud bootstrap script:

   wget https://binary.mirantis.com/releases/get_container_cloud.sh
   
   chmod 0755 get_container_cloud.sh
   
   ./get_container_cloud.sh
   

3. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

4. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

5. Save the license file as mirantis.lic under the kaas-bootstrap directory.

6. Verify access to the target cloud endpoint from Docker. For example:

   docker run --rm alpine sh -c "apk add --no-cache curl; \
   curl https://ec2.amazonaws.com"
   

   The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

7. In templates/aws/machines.yaml.template, modify the spec:providerSpec:value section by substituting the ami:id parameter with the corresponding value for Ubuntu 18.04 from the required AWS region. For example:

    spec:
      providerSpec:
        value:
          apiVersion: aws.kaas.mirantis.com/v1alpha1
          kind: AWSMachineProviderSpec
          instanceType: c5d.2xlarge
          ami:
            id: ami-033a0960d9d83ead0
   

   Also, modify other parameters as required.

8. Optional. In templates/aws/cluster.yaml.template, modify the default AWS instance types and AMIs configuration for further creation of managed clusters:

   providerSpec:
       value:
         ...
         kaas:
           ...
           regional:
           - provider: aws
             helmReleases:
               - name: aws-credentials-controller
                 values:
                   config:
                     allowedInstanceTypes:
                       minVCPUs: 8
                       # in MiB
                       minMemory: 16384
                       # in GB
                       minStorage: 120
                       supportedArchitectures:
                       - "x86_64"
                       filters:
                       - name: instance-storage-info.disk.type
                         values:
                           - "ssd"
                     allowedAMIs:
                     -
                       - name: name
                         values:
                         - "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200729"
                       - name: owner-id
                         values:
                         - "099720109477"
   

   Also, modify other parameters as required.

9. ^{Available since 2.5.0} Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

   In templates/aws/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

   spec:
     ...
     providerSpec:
       value:
         kaas:
         ...
           regional:
             - helmReleases:
               - name: aws-provider
                 values:
                   config:
                     lcm:
                       ...
                       ntp:
                         servers:
                         - 0.pool.ntp.org
                         ...
   

10. Generate the AWS Access Key ID with Secret Access Key for the admin user and select the AWS default region name. For details, see AWS General Reference: Programmatic access.

11. Export the following parameters by adding the corresponding values for the AWS admin credentials created in the previous step:

    export KAAS_AWS_ENABLED=true
    export AWS_SECRET_ACCESS_KEY=XXXXXXX
    export AWS_ACCESS_KEY_ID=XXXXXXX
    export AWS_DEFAULT_REGION=us-east-2
    

12. For Container Cloud to communicate with the AWS APIs, create the AWS CloudFormation stack that contains properly configured IAM users and policies:

    ./kaas bootstrap aws policy
    

    If you do not have access to create the CloudFormation stack, users, or policies:

    1. Log in to your AWS Management Console.

    2. On the home page, expand the upper right menu with your user name and capture your Account ID.

    3. Create the CloudFormation template:

       ./kaas bootstrap aws policy --account-id <accountId> --dump > cf.yaml
       

       Substitute the parameter enclosed in angle brackets with the corresponding value.

    4. Send the cf.yaml template to your AWS account admin to create the CloudFormation stack from this template.

13. Generate the AWS Access Key ID with Secret Access Key for the bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user, that was created in the previous step, and select the AWS default region name.

14. Export the AWS bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user credentials that were created in the previous step:

    export KAAS_AWS_ENABLED=true
    export AWS_SECRET_ACCESS_KEY=XXXXXXX
    export AWS_ACCESS_KEY_ID=XXXXXXX
    export AWS_DEFAULT_REGION=us-east-2
    

15. Optional. Skip this step to use the default password password in the Container Cloud web UI.

    Configure the IAM parameters:

    1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

       ./bin/hash-generate -i 27500
       

       The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

       Example of system response:

       passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
       passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
       passwordHashAlgorithm: pbkdf2-sha256
       passwordHashIterations: 27500
       

       Run the tool several times to generate hashed passwords for every IAM role.

    2. Open templates/cluster.yaml.template for editing.

    3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

       • passwordSalt - base64-encoded randomly generated sequence of bytes.

       • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

16. Optional. Configure external identity provider for IAM.

17. Run the bootstrap script:

    ./bootstrap.sh all
    

18. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private SSH key openstack_tmp located in ~/.ssh/ for access to the management cluster nodes.

      Note

      The SSH key name openstack_tmp is the same for all cloud providers. This name will be changed in one of the following Container Cloud releases to avoid confusion with a cloud provider name and its related SSH key name.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The StackLight endpoints. For details, see Operations Guide: Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

19. In case of deployment issues, collect and inspect the bootstrap and management cluster logs as described in Troubleshooting.

20. Optional. Deploy an additional regional cluster of a different provider type as described in Deploy an additional regional cluster (optional).

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create an AWS-based managed cluster.

Workflow overview¶

Perform the following steps to install Mirantis Container Cloud on a VMWare vSphere-based environment:

1. Prepare a vSphere environment with the requirements described in Reference Architecture: VMWare vSphere-based cluster requirements.

2. Prepare the bootstrap node as described in Prerequisites.

3. Obtain the Mirantis license file to use during the bootstrap.

4. Set up the VMWare accounts for deployment as described in VMWare deployment users.

5. Create and configure the deployment configuration files that include the cluster and machines metadata as described in Bootstrap a management cluster.

6. Prepare the OVF template for the management cluster nodes using OVF template requirements.

7. Run the bootstrap script for the fully automated installation of the management cluster.

For more details, see Bootstrap a management cluster.

Prerequisites¶

Before bootstrapping a VMWare vSphere-based management cluster, complete the following prerequisite steps:

1. Verify that your planned cloud meets the reference hardware bill of material and software requirements as described in Reference Architecture: Requirements for a VMWare vSphere-based Container Cloud cluster.

2. Configure the bootstrap node:

   1. Log in to any personal computer or VM running Ubuntu 18.04 that you will be using as the bootstrap node.

   2. If you use a newly created VM, run:

      sudo apt-get update
      

   3. Install the current Docker version available for Ubuntu 18.04:

      sudo apt install docker.io
      

   4. Grant your USER access to the Docker daemon:

      sudo usermod -aG docker $USER
      

   5. Log off and log in again to the bootstrap node to apply the changes.

   6. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://binary.mirantis.com"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

3. Prepare the VMWare deployment user setup and permissions.

Prepare the VMWare deployment user setup and permissions¶

To deploy Mirantis Container Cloud on a VMWare vSphere-based environment, prepare the following VMWare accounts:

1. Log in to the vCenter Server Web Console.

2. Create a read-only virt-who user. The virt-who user requires at least read-only access to all objects in the vCenter Data Center.

   The virt-who service on RHEL machines will be provided with the virt-who user credentials in order to properly manage RHEL subscriptions.

   For details on how to create the virt-who user, refer to the official RedHat Customer Portal documentation.

3. Create the cluster-api user with the following privileges:

   Note

   Container Cloud uses two separate vSphere accounts for:

   • Cluster API related operations, such as create or delete VMs, and for preparation of the OVF template using Packer

   • Storage operations, such as dynamic PVC provisioning

   You can also create one user that has all privileges sets mentioned above.

   Privilege	Permission
   Content library	Download files Read storage Sync library item
   Datastore	Allocate space Browse datastore Low-level file operations Update virtual machine metadata
   Folder	Create folder Rename folder
   Global	Cancel task
   Host local operations	Create virtual machine Delete virtual machine Reconfigure virtual machine
   Network	Assign network
   Resource	Assign virtual machine to resource pool
   Scheduled task	Create tasks Modify task Remove task Run task
   Sessions	Validate session View and stop sessions
   Storage views	View
   Tasks	Create task Update task

   Virtual machine permissions¶

   Privilege	Permission
   Change configuration	Acquire disk lease Add existing disk Add new disk Add or remove device Advanced configuration Change CPU count Change Memory Change Settings Change Swapfile placement Change resource Configure Host USB device Configure Raw device Configure managedBy Display connection settings Extend virtual disk Modify device settings Query Fault Tolerance compatibility Query unowned files Reload from path Remove disk Rename Reset guest information Set annotation Toggle disk change tracking Toggle fork parent Upgrade virtual machine compatibility
   Interaction	Configure CD media Configure floppy media Console interaction Device connection Inject USB HID scan codes Power off Power on Reset Suspend
   Inventory	Create from existing Create new Move Register Remove Unregister
   Provisioning	Allow disk access Allow file access Allow read-only disk access Allow virtual machine download Allow virtual machine files upload Clone template Clone virtual machine Create template from virtual machine Customize guest Deploy template Mark as template Mark as virtual machine Modify customization specification Promote disks Read customization specifications
   Snapshot management	Create snapshot Remove snapshot Rename snapshot Revert to snapshot
   vSphere replication	Monitor replication

4. Create the storage user with the following privileges:

   Note

   For more details about all required privileges for the storage user, see vSphere Cloud Provider documentation.

   Privilege	Permission
   Cloud Native Storage	Searchable
   Content library	View configuration settings
   Datastore	Allocate space Browse datastore Low level file operations Remove file
   Folder	Create folder
   Host configuration	Storage partition configuration
   Host local operations	Create virtual machine Delete virtual machine Reconfigure virtual machine
   Host profile	View
   Profile-driven storage	Profile-driven storage view
   Resource	Assign virtual machine to resource pool
   Scheduled task	Create tasks Modify task Run task
   Sessions	Validate session View and stop sessions
   Storage views	View

   Virtual machine permissions¶

   Privilege	Permission
   Change configuration	Add existing disk Add new disk Add or remove device Advanced configuration Change CPU count Change Memory Change Settings Configure managedBy Extend virtual disk Remove disk Rename
   Inventory	Create from existing Create new Remove

Now, proceed to Bootstrap a management cluster.

Bootstrap a management cluster¶

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your VMWare vSphere-based Mirantis Container Cloud management cluster.

To bootstrap a vSphere-based management cluster:

1. Log in to the bootstrap node running Ubuntu 18.04 that is configured as described in Prerequisites.

2. Download and run the Container Cloud bootstrap script:

   wget https://binary.mirantis.com/releases/get_container_cloud.sh
   chmod 0755 get_container_cloud.sh
   ./get_container_cloud.sh
   

3. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

4. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

5. Save the license file as mirantis.lic under the kaas-bootstrap directory.

6. In templates/vsphere/rhellicenses.yaml.template, set the user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters. Optionally, specify the subscription allocation pools to use for the RHEL subscriptions activation. If you leave the pool field empty, subscription-manager will automatically select the licenses for machines.

7. Modify templates/vsphere/vsphere-config.yaml.template:

   vSphere configuration data¶

   Parameter	Description
   SET_VSPHERE_SERVER	IP address or FQDN of the vCenter Server.
   SET_VSPHERE_SERVER_PORT	Port of the vCenter Server. For example, port: "8443". Leave empty to use 443 by default.
   SET_VSPHERE_DATACENTER	vSphere data center name.
   SET_VSPHERE_SERVER_INSECURE	Flag that controls validation of the vSphere Server certificate. Must be true or false.
   SET_VSPHERE_CAPI_PROVIDER_USERNAME	vSphere Cluster API provider user name. For details, see Prepare the VMWare deployment user setup and permissions.
   SET_VSPHERE_CAPI_PROVIDER_PASSWORD	vSphere Cluster API provider user password.
   SET_VSPHERE_CLOUD_PROVIDER_USERNAME	vSphere Cloud Provider deployment user name. For details, see Prepare the VMWare deployment user setup and permissions.
   SET_VSPHERE_CLOUD_PROVIDER_PASSWORD	vSphere Cloud Provider deployment user password.

8. Modify the templates/vsphere/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

   Required parameters:

   vSphere configuration data¶

   Parameter	Description
   SET_VSPHERE_DATASTORE	Name of the vSphere datastore. You can use different datastores for vSphere Cluster API and vSphere Cloud Provider.
   SET_VSPHERE_MACHINES_FOLDER	Path to a folder where the cluster machines metadata will be stored.
   SET_VSPHERE_NETWORK_PATH	Path to a network for cluster machines.
   SET_VSPHERE_RESOURCE_POOL_PATH	Path to a resource pool in which VMs will be created.

   Note

   The passwordSalt and passwordHash values for the IAM roles are automatically re-generated during the IAM configuration described below in this procedure.

9. In bootstrap.env, add the following environment variables:

   Note

   For the Keycloak and IAM services variables, assign IP addresses from the end of the provided MetalLB range. For example, if the MetalLB range is 10.20.0.30-10.20.0.50, select 10.20.0.48 and 10.20.0.49 as IPs for KeyCloak and IAM.

   vSphere environment data¶

   Parameter	Description
   KAAS_VSPHERE_ENABLED	Set to true. Enables the vSphere provider deployment in Container Cloud.
   KEYCLOAK_FLOATING_IP	IP address for Keycloak from the end of the MetalLB range.
   IAM_FLOATING_IP	IP address for IAM from the end of MetalLB range.

10. ^{Available since 2.5.0} Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

    In templates/vsphere/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

    spec:
      ...
      providerSpec:
        value:
          kaas:
          ...
            regional:
              - helmReleases:
                - name: vsphere-provider
                  values:
                    config:
                      lcm:
                        ...
                        ntp:
                          servers:
                          - 0.pool.ntp.org
                          ...
    

11. Prepare the OVF template as described in Prepare the OVF template.

12. In templates/vsphere/machines.yaml.template:

    • Define SSH_USER_NAME. The default SSH user name is cloud-user.

    • Define SET_VSPHERE_TEMPLATE_PATH prepared in the previous step.

    • Modify other parameters as required.

    spec:
      providerSpec:
        value:
          apiVersion: vsphere.cluster.k8s.io/v1alpha1
          kind: VsphereMachineProviderSpec
          sshUserName: SSH_USER_NAME
          rhelLicense: kaas-mgmt-rhel-license
          network:
            devices:
            - dhcp4: true
              dhcp6: false
          template: SET_VSPHERE_TEMPLATE_PATH
    

13. ^{Available since 2.5.0, Technology Preview} Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the management and regional cluster using proxy:

    • HTTP_PROXY

    • HTTPS_PROXY

    • NO_PROXY

    Example snippet:

    export HTTP_PROXY=http://proxy.example.com:3128
    export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
    export NO_PROXY=172.18.10.0,registry.internal.lan
    

    The following variables formats are accepted:

    vSphere configuration data¶

    Variable	Format
    HTTP_PROXY HTTPS_PROXY	http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
    NO_PROXY	Comma-separated list of IP addresses or domain names

    For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

14. Optional. Skip this step to use the default password password in the Container Cloud web UI.

    Configure the IAM parameters:

    1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

       ./bin/hash-generate -i 27500
       

       The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

       Example of system response:

       passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
       passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
       passwordHashAlgorithm: pbkdf2-sha256
       passwordHashIterations: 27500
       

       Run the tool several times to generate hashed passwords for every IAM role.

    2. Open templates/cluster.yaml.template for editing.

    3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

       • passwordSalt - base64-encoded randomly generated sequence of bytes.

       • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

15. Optional. Configure external identity provider for IAM.

16. Run the bootstrap script:

    ./bootstrap.sh all
    

17. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private SSH key openstack_tmp located in ~/.ssh/ for access to the management cluster nodes.

      Note

      The SSH key name openstack_tmp is the same for all cloud providers. This name will be changed in one of the following Container Cloud releases to avoid confusion with a cloud provider name and its related SSH key name.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The StackLight endpoints. For details, see Operations Guide: Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create a VMWare vSphere-based managed cluster.

Prepare the OVF template¶

To deploy Mirantis Container Cloud on a vSphere-based environment, the OVF template for cluster machines must be prepared according to the following requirements:

1. The VMWare Tools package is installed.

2. The cloud-init utility is installed and configured with the specific VMwareGuestInfo data source.

3. The virt-who service is enabled and configured to connect to the VMWare vCenter Server to properly apply the RHEL subscriptions on the nodes.

The following procedures describe how to meet the requirements above either using the Container Cloud script or manually.

To prepare the OVF template using the Container Cloud script:

1. Prepare the Container Cloud bootstrap and modify vsphere-config.yaml.template and templates/vsphere/cluster.yaml.template as described in Bootstrap a management cluster, steps 1-9.

2. Download the RHEL 7.8 DVD ISO from the RedHat Customer Portal.

3. Export the following variables:

   1. The virt-who user name and password.

   2. The path to the RHEL 7.8 DVD ISO file.

   3. The vSphere cluster name.

   For example:

   export KAAS_VSPHERE_ENABLED=true
   export VSPHERE_RO_USER=virt-who-user
   export VSPHERE_RO_PASSWORD=virt-who-user-password
   export VSPHERE_PACKER_ISO_FILE=$(pwd)/rhel-7.8.dvd.iso
   export VSPHERE_CLUSTER_NAME=vsphere-cluster-name
   

4. ^{Available since 2.5.0, Technology Preview} Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

   Example snippet:

   export HTTP_PROXY=http://proxy.example.com:3128
   export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
   export NO_PROXY=172.18.10.0,registry.internal.lan
   

   The following variables formats are accepted:

   vSphere configuration data¶

   Variable	Format
   HTTP_PROXY HTTPS_PROXY	http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
   NO_PROXY	Comma-separated list of IP addresses or domain names

   For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

5. Prepare the OVF template:

   ./bootstrap.sh vsphere_template
   

6. After the template is prepared, set the SET_VSPHERE_TEMPLATE_PATH parameter in templates/vsphere/machines.yaml.template as described in Bootstrap a management cluster.

To prepare the OVF template manually:

1. Run a virtual machine on the vSphere data center from the official RHEL 7.8 server image. Specify the amount of resources that will be used in the Container Cloud setup. A minimal resources configuration must match the requirements for a vSphere-based Container Cloud cluster.

2. Select minimal setup in the VM installation configuration. Create a user with root or sudo permissions to access the machine.

3. Log in to the VM when it starts.

4. ^{Available since 2.5.0, Technology Preview} Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

   Example snippet:

   export HTTP_PROXY=http://proxy.example.com:3128
   export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
   export NO_PROXY=172.18.10.0,registry.internal.lan
   

   The following variables formats are accepted:

   vSphere configuration data¶

   Variable	Format
   HTTP_PROXY HTTPS_PROXY	http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
   NO_PROXY	Comma-separated list of IP addresses or domain names

   For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

5. Attach your RHEL license for Virtual Datacenters to the VM:

   subscription-manager register
   # automatic subscription selection:
   subscription-manager attach --auto
   # or specify pool id:
   subscription-manager attach --pool=<POOL_ID>
   # verify subscription status
   subscription-manager status
   

6. Select from the following options:

   • Prepare the operating system automatically:

     1. Download the automation script:

        curl https://gerrit.mcp.mirantis.com/plugins/gitiles/kubernetes/vmware-guestinfo/+/refs/tags/  v1.1.1/install.sh?format=TEXT | \
        base64 -d > install.sh
        chmod +x install.sh
        

     2. Export the vCenter Server credentials of the read-only user. For example:

        export VC_SERVER='vcenter1.example.com'
        export VC_USER='domain\vmware_read_only_username'
        export VC_PASSWORD='password!23'
        # optional parameters:
        export VC_HYPERVISOR_ID=hostname
        export VC_FILTER_HOSTS="esx1.example.com, esx2.example.com"
        export VCENTER_CONFIG_PATH="/etc/virt-who.d/vcenter.conf"
        

     3. Run the installation script:

        ./install.sh
        

   • Prepare the operating system manually:

     1. Install open-vm-tools:

        yum install open-vm-tools -y
        

     2. Install and configure cloud-init:

        1. Download the VMwareGuestInfo data source files:

           curl https://gerrit.mcp.mirantis.com/plugins/gitiles/kubernetes/vmware-guestinfo/+/refs/tags/v1.1.1/DataSourceVMwareGuestInfo.py?format=TEXT | \
           base64 -d > DataSourceVMwareGuestInfo.py
           curl https://gerrit.mcp.mirantis.com/plugins/gitiles/kubernetes/vmware-guestinfo/+/refs/tags/v1.1.1/99-DataSourceVMwareGuestInfo.cfg?format=TEXT | \
           base64 -d > 99-DataSourceVMwareGuestInfo.cfg
           

        2. Add 99-DataSourceVMwareGuestInfo.cfg to /etc/cloud/cloud.cfg.d/.

        3. Depending on the Python version on the VM operating system, add DataSourceVMwareGuestInfo.py to the cloud-init sources folder.

        4. Obtain the cloud-init folder on RHEL:

           yum install cloud-init -y
           python -c 'import os; from cloudinit import sources; print(os.path.dirname(sources.__file__));'
           

     3. Prepare the virt-who user configuration:

        Note

        For details about the virt-who user creation, see Prepare the VMWare deployment user setup and permissions.

        1. Install virt-who:

           yum install virt-who -y
           cp /etc/virt-who.d/template.conf /etc/virt-who.d/vcenter.conf
           

        2. Set up the file content using the following example:

           [vcenter]
           type=esx
           server=vcenter1.example.com
           username=domain\vmware_read_only_username
           encrypted_password=bd257f93d@482B76e6390cc54aec1a4d
           owner=1234567
           hypervisor_id=hostname
           filter_hosts=esx1.example.com, esx2.example.com
           

           virt-who configuration parameters¶

           Parameter	Description
           [vcenter]	Name of the vCenter data center.
           type=esx	Specifies the connection of the defined virt-who user to the vCenter Server.
           server	The FQDN of the vCenter Server.
           username	The virt-who user name on the vCenter Server with the read-only access.
           encrypted_password	The virt-who password encrypted by the virt-who-password utility using the virt-who-password -p <password> command.
           owner	The organization that the hypervisors belong to.
           hypervisor_id	Specifies how to identify the hypervisors. Use a host name to provide meaningful host names to the Subscription Management. Alternatively, use uuid or hwuuid to avoid duplication in case of hypervisor renaming.
           filter_hosts	List of hypervisors that never run RHEL VMs. Such hypervisors do not have to be reported by virt-who.

7. Remove the RHEL subscription from the node.

   subscription-manager remove --all
   subscription-manager unregister
   subscription-manager clean
   

8. Shut down the VM.

9. Create an OVF template from the VM.

Now, proceed to Bootstrap a management cluster.

Deploy an AWS-based regional cluster¶

If you want to deploy AWS-based managed clusters of different configurations, deploy an additional regional cluster with specific settings that differ from the AWS-based management cluster configuration.

To deploy an AWS-based regional cluster:

1. Log in to the node where you bootstrapped a management cluster.

2. Prepare the AWS configuration for the new regional cluster:

   1. Verify access to the target cloud endpoint from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://ec2.amazonaws.com"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

   2. Change the directory to the kaas-bootstrap folder.

   3. In templates/aws/machines.yaml.template, modify the spec:providerSpec:value section by substituting the ami:id parameter with the corresponding value for Ubuntu 18.04 from the required AWS region. For example:

       spec:
         providerSpec:
           value:
             apiVersion: aws.kaas.mirantis.com/v1alpha1
             kind: AWSMachineProviderSpec
             instanceType: c5d.2xlarge
             ami:
               id: ami-033a0960d9d83ead0
      

      Also, modify other parameters as required.

   4. Optional. In templates/aws/cluster.yaml.template, modify the default configuration of the AWS instance types and AMI IDs for further creation of managed clusters:

      providerSpec:
          value:
            ...
            kaas:
              ...
              regional:
              - provider: aws
                helmReleases:
                  - name: aws-credentials-controller
                    values:
                      config:
                        allowedInstanceTypes:
                          minVCPUs: 8
                          # in MiB
                          minMemory: 16384
                          # in GB
                          minStorage: 120
                          supportedArchitectures:
                          - "x86_64"
                          filters:
                          - name: instance-storage-info.disk.type
                            values:
                              - "ssd"
                        allowedAMIs:
                        -
                          - name: name
                            values:
                            - "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200729"
                          - name: owner-id
                            values:
                            - "099720109477"
      

      Also, modify other parameters as required.

3. ^{Available since 2.5.0} Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

   In templates/aws/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

   spec:
     ...
     providerSpec:
       value:
         kaas:
         ...
           regional:
             - helmReleases:
               - name: aws-provider
                 values:
                   config:
                     lcm:
                       ...
                       ntp:
                         servers:
                         - 0.pool.ntp.org
                         ...
   

   1. Generate the AWS Access Key ID with Secret Access Key for the bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user and select the AWS default region name.

   2. Export the AWS bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user credentials that were created in the previous step:

      export KAAS_AWS_ENABLED=true
      export AWS_SECRET_ACCESS_KEY=XXXXXXX
      export AWS_ACCESS_KEY_ID=XXXXXXX
      export AWS_DEFAULT_REGION=us-east-2
      

4. Export the following parameters:

   export KUBECONFIG=<pathToMgmtClusterKubeconfig>
   export REGIONAL_CLUSTER_NAME=<newRegionalClusterName>
   export REGION=<NewRegionName>
   

   Substitute the parameters enclosed in angle brackets with the corresponding values of your cluster.

5. Run the regional cluster bootstrap script:

   ./bootstrap.sh deploy_regional
   

   Note

   When the bootstrap is complete, obtain and save in a secure location the kubeconfig-<regionalClusterName> file located in the same directory as the bootstrap script. This file contains the admin credentials for the regional cluster.

   The workflow of the regional cluster bootstrap script¶

   #	Description
   1	Prepare the bootstrap cluster for the new regional cluster.
   2	Load the updated Container Cloud CRDs for Credentials, Cluster, and Machines with information about the new regional cluster to the management cluster.
   3	Connect to each machine of the management cluster through SSH.
   4	Wait for the Machines and Cluster objects of the new regional cluster to be ready on the management cluster.
   5	Load the following objects to the new regional cluster: Secret with the management cluster kubeconfig and ClusterRole for the Container Cloud provider.
   6	Forward the bootstrap cluster endpoint to helm-controller.
   7	Wait for all CRDs to be available and verify the objects created using these CRDs.
   8	Pivot the cluster API stack to the regional cluster.
   9	Switch the LCM agent from the bootstrap cluster to the regional one.
   10	Wait for the Container Cloud components to start on the regional cluster.

Now, you can proceed with deploying the managed clusters of supported provider types as described in Create and operate a managed cluster.

Deploy an OpenStack-based regional cluster¶

You can deploy an additional regional OpenStack-based cluster on top of the AWS, bare metal, or OpenStack management cluster to create managed clusters of several provider types if required.

To deploy an OpenStack-based regional cluster:

1. Log in to the node where you bootstrapped a management cluster.

2. Prepare the OpenStack configuration for a new regional cluster:

   1. Log in to the OpenStack Horizon.

   2. In the Project section, select API Access.

   3. In the right-side drop-down menu Download OpenStack RC File, select OpenStack clouds.yaml File.

   4. Add the downloaded clouds.yaml file to the directory with the bootstrap.sh script.

   5. In clouds.yaml, add the password field with your OpenStack password under the clouds/openstack/auth section.

      Example:

      clouds:
        openstack:
          auth:
            auth_url: https://auth.openstack.example.com:5000/v3
            username: your_username
            password: your_secret_password
            project_id: your_project_id
            user_domain_name: your_user_domain_name
          region_name: RegionOne
          interface: public
          identity_api_version: 3
      

   6. Verify access to the target cloud endpoint from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://auth.openstack.example.com:5000/v3"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

3. Configure the cluster and machines metadata:

   1. Change the directory to the kaas-bootstrap folder.

   2. In templates/machines.yaml.template, modify the spec:providerSpec:value section for 3 control plane nodes marked with the cluster.sigs.k8s.io/control-plane label by substituting the flavor and image parameters with the corresponding values of the control plane nodes in the related OpenStack cluster. For example:

      spec: &cp_spec
        providerSpec:
          value:
            apiVersion: "openstackproviderconfig.k8s.io/v1alpha1"
            kind: "OpenstackMachineProviderSpec"
            flavor: kaas.minimal
            image: bionic-server-cloudimg-amd64-20190612
      

      Also, modify other parameters as required.

   3. Modify the templates/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

   4. ^{Available since 2.5.0} Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

      In templates/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

      spec:
        ...
        providerSpec:
          value:
            kaas:
            ...
              regional:
                - helmReleases:
                  - name: openstack-provider
                    values:
                      config:
                        lcm:
                          ...
                          ntp:
                            servers:
                            - 0.pool.ntp.org
                            ...
      

4. ^{Available since 2.5.0} Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the regional cluster using proxy:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

   Example snippet:

   export HTTP_PROXY=http://proxy.example.com:3128
   export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
   export NO_PROXY=172.18.10.0,registry.internal.lan
   

   The following variables formats are accepted:

   vSphere configuration data¶

   Variable	Format
   HTTP_PROXY HTTPS_PROXY	http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
   NO_PROXY	Comma-separated list of IP addresses or domain names

   For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

5. Clean up the environment configuration:

   1. If you are deploying the regional cluster on top of a baremetal-based management cluster, unset the following parameters:

      unset KAAS_BM_ENABLED KAAS_BM_FULL_PREFLIGHT KAAS_BM_PXE_IP \
            KAAS_BM_PXE_MASK KAAS_BM_PXE_BRIDGE KAAS_BM_BM_DHCP_RANGE \
            TEMPLATES_DIR
      

   2. If you are deploying the regional cluster on top of an AWS-based management cluster, unset the KAAS_AWS_ENABLED parameter:

      unset KAAS_AWS_ENABLED
      

6. Export the following parameters:

   export KUBECONFIG=<pathToMgmtClusterKubeconfig>
   export REGIONAL_CLUSTER_NAME=<newRegionalClusterName>
   export REGION=<NewRegionName>
   

   Substitute the parameters enclosed in angle brackets with the corresponding values of your cluster.

7. Run the regional cluster bootstrap script:

   ./bootstrap.sh deploy_regional
   

   Note

   When the bootstrap is complete, obtain and save in a secure location the kubeconfig-<regionalClusterName> file located in the same directory as the bootstrap script. This file contains the admin credentials for the regional cluster.

   The workflow of the regional cluster bootstrap script¶

   #	Description
   1	Prepare the bootstrap cluster for the new regional cluster.
   2	Load the updated Container Cloud CRDs for Credentials, Cluster, and Machines with information about the new regional cluster to the management cluster.
   3	Connect to each machine of the management cluster through SSH.
   4	Wait for the Machines and Cluster objects of the new regional cluster to be ready on the management cluster.
   5	Load the following objects to the new regional cluster: Secret with the management cluster kubeconfig and ClusterRole for the Container Cloud provider.
   6	Forward the bootstrap cluster endpoint to helm-controller.
   7	Wait for all CRDs to be available and verify the objects created using these CRDs.
   8	Pivot the cluster API stack to the regional cluster.
   9	Switch the LCM agent from the bootstrap cluster to the regional one.
   10	Wait for the Container Cloud components to start on the regional cluster.

Now, you can proceed with deploying the managed clusters of supported provider types as described in Create and operate a managed cluster.

Deploy a VMWare vSphere-based regional cluster¶

You can deploy an additional regional VMWare vSphere-based cluster on top of the AWS or vSphere management cluster to create managed clusters with different configurations if required.

To deploy a vSphere-based regional cluster:

1. Log in to the node where you bootstrapped a management cluster.

2. Prepare the vSphere configuration for the new regional cluster:

   1. Verify access to the target vSphere cluster from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://vsphere.server.com"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

   2. In templates/vsphere/rhellicenses.yaml.template, set the user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters. Optionally, specify the subscription allocation pools to use for the RHEL subscriptions activation. If you leave the pool field empty, subscription-manager will automatically select the licenses for machines.

   3. Modify the templates/vsphere/cluster.yaml.template parameters to fit your deployment.

      Required parameters:

      vSphere configuration data¶

      Parameter	Description
      SET_VSPHERE_DATASTORE	vSphere datastore name. You can use different datastores for vSphere Cluster API and vSphere Cloud Provider.
      SET_VSPHERE_MACHINES_FOLDER	Path to a folder where the cluster machines metadata will be stored.
      SET_VSPHERE_NETWORK_PATH	Path to a network for cluster machines.
      SET_VSPHERE_RESOURCE_POOL_PATH	Path to a resource pool in which VMs will be created.

      Note

      The passwordSalt and passwordHash values for the IAM roles are automatically re-generated during the IAM configuration described below in this procedure.

3. ^{Available since 2.5.0} Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

   In templates/vsphere/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

   spec:
     ...
     providerSpec:
       value:
         kaas:
         ...
           regional:
             - helmReleases:
               - name: vsphere-provider
                 values:
                   config:
                     lcm:
                       ...
                       ntp:
                         servers:
                         - 0.pool.ntp.org
                         ...
   

   1. Modify templates/vsphere/vsphere-config.yaml.template:

      vSphere configuration data¶

      Parameter	Description
      SET_VSPHERE_SERVER	IP address or FQDN of the vCenter Server.
      SET_VSPHERE_SERVER_PORT	Port of the vCenter Server. Leave empty to use 443 by default.
      SET_VSPHERE_DATACENTER	vSphere data center name.
      SET_VSPHERE_SERVER_INSECURE	Flag that controls validation of the vSphere Server certificate.
      SET_VSPHERE_CAPI_PROVIDER_USERNAME	vSphere Cluster API provider user name. For details, see Prepare the VMWare deployment user setup and permissions.
      SET_VSPHERE_CAPI_PROVIDER_PASSWORD	vSphere Cluster API provider user password.
      SET_VSPHERE_CLOUD_PROVIDER_USERNAME	vSphere Cloud Provider deployment user name. For details, see Prepare the VMWare deployment user setup and permissions.
      SET_VSPHERE_CLOUD_PROVIDER_PASSWORD	vSphere Cloud Provider deployment user password.

   2. Prepare the OVF template as described in Prepare the OVF template.

   3. In templates/vsphere/machines.yaml.template:

      • Define SSH_USER_NAME. The default SSH user name is cloud-user.

      • Define SET_VSPHERE_TEMPLATE_PATH prepared in the previous step.

      • Modify other parameters as required.

      spec:
        providerSpec:
          value:
            apiVersion: vsphere.cluster.k8s.io/v1alpha1
            kind: VsphereMachineProviderSpec
            sshUserName: SSH_USER_NAME
            rhelLicense: kaas-mgmt-rhel-license
            network:
              devices:
              - dhcp4: true
                dhcp6: false
            template: SET_VSPHERE_TEMPLATE_PATH
      

4. ^{Available since 2.5.0, Technology Preview} Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the regional cluster using proxy:

   • HTTP_PROXY

   • HTTPS_PROXY

   • NO_PROXY

   Example snippet:

   export HTTP_PROXY=http://proxy.example.com:3128
   export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
   export NO_PROXY=172.18.10.0,registry.internal.lan
   

   The following variables formats are accepted:

   vSphere configuration data¶

   Variable	Format
   HTTP_PROXY HTTPS_PROXY	http://proxy.example.com:port - for anonymous access http://user:password@proxy.example.com:port - for restricted access
   NO_PROXY	Comma-separated list of IP addresses or domain names

   For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

5. Export the following parameters:

   export KAAS_VSPHERE_ENABLED=true
   export KUBECONFIG=<pathToMgmtClusterKubeconfig>
   export REGIONAL_CLUSTER_NAME=<newRegionalClusterName>
   export REGION=<NewRegionName>
   

   Substitute the parameters enclosed in angle brackets with the corresponding values of your cluster.

6. Run the regional cluster bootstrap script:

   ./bootstrap.sh deploy_regional
   

   Note

   When the bootstrap is complete, obtain and save in a secure location the kubeconfig-<regionalClusterName> file located in the same directory as the bootstrap script. This file contains the admin credentials for the regional cluster.

   The workflow of the regional cluster bootstrap script¶

   #	Description
   1	Prepare the bootstrap cluster for the new regional cluster.
   2	Load the updated Container Cloud CRDs for Credentials, Cluster, and Machines with information about the new regional cluster to the management cluster.
   3	Connect to each machine of the management cluster through SSH.
   4	Wait for the Machines and Cluster objects of the new regional cluster to be ready on the management cluster.
   5	Load the following objects to the new regional cluster: Secret with the management cluster kubeconfig and ClusterRole for the Container Cloud provider.
   6	Forward the bootstrap cluster endpoint to helm-controller.
   7	Wait for all CRDs to be available and verify the objects created using these CRDs.
   8	Pivot the cluster API stack to the regional cluster.
   9	Switch the LCM agent from the bootstrap cluster to the regional one.
   10	Wait for the Container Cloud components to start on the regional cluster.

Now, you can proceed with deploying the managed clusters of supported provider types as described in Create and operate a managed cluster.

Collect the bootstrap logs¶

If the bootstrap script fails during the deployment process, collect and inspect the bootstrap and management cluster logs.

To collect the bootstrap logs:

1. Log in to your local machine where the bootstrap script was executed.

2. Run the following command:

   ./bootstrap.sh collect_logs
   

   The logs are collected in the directory where the bootstrap script is located.

The Container Cloud logs structure in <output_dir>/<cluster_name>/ is as follows:

• /events.log - human-readable table that contains information about the cluster events

• /system - system logs

• /system/<machine_name>/ucp - Mirantis Kuberntes Engine (MKE) logs

• /objects/cluster - logs of the non-namespaced Kubernetes objects

• /objects/namespaced - logs of the namespaced Kubernetes objects

• /objects/namespaced/<namespaceName>/core/pods - pods logs from a specified Kubernetes namespace

• /objects/namespaced/<namespaceName>/core/pods/<containerName>.prev.log - logs of the pods from a specified Kubernetes namespace that were previously removed or failed

Depending on the type of issue found in logs, apply the corresponding fixes. For example, if you detect the LoadBalancer ERROR state errors during the bootstrap of an OpenStack-based management cluster, contact your system administrator to fix the issue. To troubleshoot other issues, refer to the corresponding section in Troubleshooting.

DNS settings¶

If you have issues related to the DNS settings, the following error message may occur:

curl: (6) Could not resolve host

The issue may occur if a VPN is used to connect to the cloud or a local DNS forwarder is set up.

The workaround is to change the default DNS settings for Docker:

1. Log in to your local machine.

2. Identify your internal or corporate DNS server address:

   systemd-resolve --status
   

3. Create or edit /etc/docker/daemon.json by specifying your DNS address:

   {
     "dns": ["<YOUR_DNS_ADDRESS>"]
   }
   

4. Restart the Docker daemon:

   sudo systemctl restart docker
   

Default network address¶

If you have issues related to the default network address configuration, cURL either hangs or the following error occurs:

curl: (7) Failed to connect to xxx.xxx.xxx.xxx port xxxx: Host is unreachable

The issue may occur because the default Docker network address 172.17.0.0/16 overlaps with your cloud address or other addresses of the network configuration.

Workaround:

1. Log in to your local machine.

2. Verify routing to the IP addresses of the target cloud endpoints:

   1. Obtain the IP address of your target cloud. For example:

      nslookup auth.openstack.example.com
      

      Example of system response:

      Name:   auth.openstack.example.com
      Address: 172.17.246.119
      

   2. Verify that this IP address is not routed through docker0 but through any other interface, for example, ens3:

      ip r get 172.17.246.119
      

      Example of the system response if the routing is configured correctly:

      172.17.246.119 via 172.18.194.1 dev ens3 src 172.18.1.1 uid 1000
        cache
      

      Example of the system response if the routing is configured incorrectly:

      172.17.246.119 via 172.18.194.1 dev docker0 src 172.18.1.1 uid 1000
        cache
      

3. If the routing is incorrect, change the IP address of the default Docker bridge:

   1. Create or edit /etc/docker/daemon.json by adding the "bip" option:

      {
        "bip": "192.168.91.1/24"
      }
      

   2. Restart the Docker daemon:

      sudo systemctl restart docker
      

TLS handshake timeout¶

If you execute the bootstrap.sh script from an OpenStack VM that is running on the OpenStack environment used for bootstrapping the management cluster, the following error messages may occur that can be related to the MTU settings discrepancy:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to server:port

Failed to check if machine "<machine_name>" exists:
failed to create provider client ... TLS handshake timeout

To identify whether the issue is MTU-related:

1. Log in to the OpenStack VM in question.

2. Compare the MTU outputs for the docker0 and ens3 interfaces:

   ip addr
   

   Example of system response:

   3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500...
   ...
   2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450...
   

   If the MTU output values differ for docker0 and ens3, proceed with the workaround below. Otherwise, inspect the logs further to identify the root cause of the error messages.

Workaround:

1. In your OpenStack environment used for Mirantis Container Cloud, log in to any machine with CLI access to OpenStack. For example, you can create a new Ubuntu VM (separate from the bootstrap VM) and install the python-openstackclient package on it.

2. Change the vXLAN MTU size for the VM to the required value depending on your network infrastructure and considering your physical network configuration, such as Jumbo frames, and so on.

   openstack network set --mtu <YOUR_MTU_SIZE> <network-name>
   

3. Stop and start the VM in Nova.

4. Log in to the bootstrap VM dedicated for the management cluster.

5. Re-execute the bootstrap.sh script.

Configure LDAP for IAM¶

If you integrate LDAP for IAM to Mirantis Container Cloud, add the required LDAP configuration to cluster.yaml.template during the bootstrap of the management cluster.

authType: "simple"
bindCredential: ""
bindDn: ""

To configure LDAP for IAM:

1. Select from the following options:

   • For a baremetal-based management cluster, open the templates/bm/cluster.yaml.template file for editing.

   • For an OpenStack management cluster, open the templates/cluster.yaml.template file for editing.

   • For an AWS-based management cluster, open the templates/aws/cluster.yaml.template file for editing.

2. Configure the keycloak:userFederation:providers: and keycloak:userFederation:mappers: sections as required:

   Note

   • Verify that the userFederation section is located on the same level as the initUsers section.

   • Verify that all attributes set in the mappers section are defined for users in the specified LDAP system. Missing attributes may cause authorization issues.

   spec:
     providerSpec:
       value:
         kaas:
           management:
             helmReleases:
             - name: iam
               values:
                 keycloak:
                   userFederation:
                     providers:
                       - displayName: "<LDAP_NAME>"
                         providerName: "ldap"
                         priority: 1
                         fullSyncPeriod: -1
                         changedSyncPeriod: -1
                         config:
                           pagination: "true"
                           debug: "false"
                           searchScope: "1"
                           connectionPooling: "true"
                           usersDn: "<DN>" # "ou=People, o=<ORGANIZATION>, dc=<DOMAIN_COMPONENT>"
                           userObjectClasses: "inetOrgPerson,organizationalPerson"
                           usernameLDAPAttribute: "uid"
                           rdnLDAPAttribute: "uid"
                           vendor: "ad"
                           editMode: "READ_ONLY"
                           uuidLDAPAttribute: "uid"
                           connectionUrl: "ldap://<LDAP_DNS>"
                           syncRegistrations: "false"
                           authType: "simple"
                           bindCredential: ""
                           bindDn: ""
                     mappers:
                       - name: "username"
                         federationMapperType: "user-attribute-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.attribute: "uid"
                           user.model.attribute: "username"
                           is.mandatory.in.ldap: "true"
                           read.only: "true"
                           always.read.value.from.ldap: "false"
                       - name: "full name"
                         federationMapperType: "full-name-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.full.name.attribute: "cn"
                           read.only: "true"
                           write.only: "false"
                       - name: "last name"
                         federationMapperType: "user-attribute-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.attribute: "sn"
                           user.model.attribute: "lastName"
                           is.mandatory.in.ldap: "true"
                           read.only: "true"
                           always.read.value.from.ldap: "true"
                       - name: "email"
                         federationMapperType: "user-attribute-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.attribute: "mail"
                           user.model.attribute: "email"
                           is.mandatory.in.ldap: "false"
                           read.only: "true"
                           always.read.value.from.ldap: "true"
   

Now, return to the bootstrap instruction depending on the provider type of your management cluster.

Configure Google OAuth IdP for IAM¶

If you integrate Google OAuth external identity provider for IAM to Mirantis Container Cloud, create the authorization credentials for IAM in your Google OAuth account and configure cluster.yaml.template during the bootstrap of the management cluster.

To configure Google OAuth IdP for IAM:

1. Create Google OAuth credentials for IAM:

   1. Log in to your https://console.developers.google.com.

   2. Navigate to Credentials.

   3. In the APIs Credentials menu, select OAuth client ID.

   4. In the window that opens:

      1. In the Application type menu, select Web application.

      2. In the Authorized redirect URIs field, type in <keycloak-url>/auth/realms/iam/broker/google/endpoint, where <keycloak-url> is the corresponding DNS address.

      3. Press Enter to add the URI.

      4. Click Create.

      A page with your client ID and client secret opens. Save these credentials for further usage.

2. Log in to the bootstrap node.

3. Select from the following options:

   • For a baremetal-based management cluster, open the templates/bm/cluster.yaml.template file for editing.

   • For an OpenStack management cluster, open the templates/cluster.yaml.template file for editing.

   • For an AWS-based management cluster, open the templates/aws/cluster.yaml.template file for editing.

4. In the keycloak:externalIdP: section, add the following snippet with your credentials created in previous steps:

   keycloak:
     externalIdP:
       google:
         enabled: true
         config:
           clientId: <Google_OAuth_client_ID>
           clientSecret: <Google_OAuth_client_secret>
   

Now, return to the bootstrap instruction depending on the provider type of your management cluster.