Mirantis Container Cloud Deployment Guide¶

Workflow overview¶

The bare metal management system enables the Infrastructure Operator to deploy Mirantis Container Cloud on a set of bare metal servers. It also enables Container Cloud to deploy managed clusters on bare metal servers without a pre-provisioned operating system.

The Infrastructure Operator performs the following steps to install Container Cloud in a bare metal environment:

1. Install and connect hardware servers as described in Reference Architecture: Baremetal-based Container Cloud cluster.

   Caution

   The baremetal-based Container Cloud does not manage the underlay networking fabric but requires specific network configuration to operate.

2. Install Ubuntu 18.04 on one of the bare metal machines to create a seed node and copy the bootstrap tarball to this node.

3. Obtain the Mirantis license file that will be required during the bootstrap.

4. Create the deployment configuration files that include the bare metal hosts metadata.

5. Validate the deployment templates using fast preflight or simulate the main stages of the management cluster deployment using full preflight.

6. Run the bootstrap script for the fully automated installation of the management cluster onto the selected bare metal hosts.

Using the bootstrap script, the Container Cloud bare metal management system prepares the seed node for the management cluster and starts the deployment of Container Cloud itself. The bootstrap script performs all necessary operations to perform the automated management cluster setup. The deployment diagram below illustrates the bootstrap workflow of a baremetal-based management cluster.

Bootstrap a management cluster¶

This section describes how to prepare and bootstrap a baremetal-based management cluster. The procedure includes:

• A runbook that describes how to create a seed node that is a temporary server used to run the management cluster bootstrap scripts.

• A step-by-step instruction how to prepare metadata for the bootstrap scripts and how to run them.

Customize the default bare metal host profile¶

This section describes the bare metal host profile settings and instructs how to configure this profile before deploying Mirantis Container Cloud on physical servers.

The bare metal host profile is a Kubernetes custom resource. It allows the Infrastructure Operator to define how the storage devices and the operating system are provisioned and configured.

The bootstrap templates for a bare metal deployment include the template for the default BareMetalHostProfile object in the following file that defines the default bare metal host profile:

templates/bm/baremetalhostprofiles.yaml.template

The customization procedure of BareMetalHostProfile is almost the same for the management and managed clusters, with the following differences:

• For a management cluster, the customization automatically applies to machines during bootstrap. And for a managed cluster, you apply the changes using kubectl before creating a managed cluster.

• For a management cluster, you edit the default baremetalhostprofiles.yaml.template. And for a managed cluster, you create a new BareMetalHostProfile with the necessary configuration.

For the procedure details, see Operations Guide: Create a custom bare metal host profile. Use this procedure for both types of clusters considering the differences described above.

Workflow overview¶

The Infrastructure Operator performs the following steps to install Mirantis Container Cloud on an OpenStack-based environment:

1. Prepare an OpenStack environment with the requirements described in Reference Architecture: OpenStack-based cluster requirements.

2. Prepare the bootstrap node using Prerequisites.

3. Obtain the Mirantis license file that will be required during the bootstrap.

4. Prepare the OpenStack clouds.yaml file.

5. Create and configure the deployment configuration files that include the cluster and machines metadata.

6. Run the bootstrap script for the fully automated installation of the management cluster.

For more details, see Bootstrap a management cluster.

Prerequisites¶

Before you start with bootstrapping the OpenStack-based management cluster, complete the following prerequisite steps:

1. Verify that your planned cloud meets the reference hardware bill of material and software requirements as described in Reference Architecture: Requirements for an OpenStack-based Mirantis Container Cloud.

2. Log in to any personal computer or VM running Ubuntu 18.04 that you will be using as the bootstrap node.

3. Install Docker version 18.09:

   sudo apt install docker.io
   

4. Grant your USER access to the Docker daemon:

   sudo usermod -aG docker $USER
   

5. Log off and log in again to the bootstrap node to apply the changes.

6. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

   docker run --rm alpine sh -c "apk add --no-cache curl; \
   curl https://binary.mirantis.com"
   

   The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

7. Proceed to Bootstrap a management cluster.

Bootstrap a management cluster¶

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your OpenStack-based Mirantis Container Cloud management cluster.

To bootstrap an OpenStack-based management cluster:

1. Log in to the bootstrap node running Ubuntu 18.04 that is configured as described in Prerequisites.

2. Download and run the Container Cloud bootstrap script:

   wget https://binary.mirantis.com/releases/get_container_cloud.sh
   chmod 0755 get_container_cloud.sh
   ./get_container_cloud.sh
   

3. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

4. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

5. Save the license file as mirantis.lic under the kaas-bootstrap directory.

6. Log in to the OpenStack Horizon.

7. In the Project section, select API Access.

8. In the right-side drop-down menu Download OpenStack RC File, select OpenStack clouds.yaml File.

9. Add the downloaded clouds.yaml file to the directory with the bootstrap.sh script.

10. In clouds.yaml, add the password field with your OpenStack password under the clouds/openstack/auth section.

    Example:

    clouds:
      openstack:
        auth:
          auth_url: https://auth.openstack.example.com:5000/v3
          username: your_username
          password: your_secret_password
          project_id: your_project_id
          user_domain_name: your_ldap_password
        region_name: RegionOne
        interface: public
        identity_api_version: 3
    

11. Verify access to the target cloud endpoint from Docker. For example:

    docker run --rm alpine sh -c "apk add --no-cache curl; \
    curl https://auth.openstack.example.com:5000/v3"
    

    The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

12. In templates/machines.yaml.template, modify the spec:providerSpec:value sections for set: master and set: node by substituting the flavor and image parameters with the corresponding values of your OpenStack cluster. For example:

    spec:
      providerSpec:
        value:
          apiVersion: "openstackproviderconfig.k8s.io/v1alpha1"
          kind: "OpenstackMachineProviderSpec"
          flavor: kaas.minimal
          image: bionic-server-cloudimg-amd64-20190612
    

    Also, modify other parameters as required.

13. Modify the templates/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

    Note

    The passwordSalt and passwordHash values for the IAM roles are automatically re-generated during the IAM configuration described in the next step.

14. Optional. Skip this step to use the default password password in the Container Cloud web UI.

    Configure the IAM parameters:

    1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

       ./bin/hash-generate -i 27500
       

       The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

       Example of system response:

       passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
       passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
       passwordHashAlgorithm: pbkdf2-sha256
       passwordHashIterations: 27500
       

       Run the tool several times to generate hashed passwords for every IAM role.

    2. Open templates/cluster.yaml.template for editing.

    3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

       • passwordSalt - base64-encoded randomly generated sequence of bytes.

       • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

15. Optional. Configure external identity provider for IAM.

16. Run the bootstrap script:

    ./bootstrap.sh all
    

17. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private SSH key openstack_tmp located in ~/.ssh/ for access to the management cluster nodes.

      Note

      The SSH key name openstack_tmp is the same for all cloud providers. This name will be changed in one of the following Container Cloud releases to avoid confusion with a cloud provider name and its related SSH key name.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

18. In case of deployment issues, collect and inspect the bootstrap and management cluster logs as described in Troubleshooting.

19. Optional. Deploy an additional regional cluster as described in Deploy an additional regional cluster.

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create an OpenStack-based managed cluster.

Workflow overview¶

The Infrastructure Operator performs the following steps to install Mirantis Container Cloud on an AWS-based environment:

1. Prepare an AWS environment with the requirements described in Reference Architecture: AWS-based Container Cloud cluster requirements.

2. Prepare the bootstrap node as per Prerequisites.

3. Obtain the Mirantis license file that will be required during the bootstrap.

4. Prepare the AWS environment credentials.

5. Create and configure the deployment configuration files that include the cluster and machines metadata.

6. Run the bootstrap script for the fully automated installation of the management cluster.

For more details, see Bootstrap a management cluster.

Prerequisites¶

Before you start with bootstrapping the AWS-based management cluster, complete the following prerequisite steps:

1. Inspect the Requirements for an AWS-based Container Cloud cluster to understand the potential impact of the Container Cloud deployment on your AWS cloud usage.

2. Log in to any personal computer or VM running Ubuntu 18.04 that you will be using as the bootstrap node.

3. If you use a newly created VM, run:

   sudo apt-get update
   

4. Install Docker version 18.09:

   sudo apt install docker.io
   

5. Grant your USER access to the Docker daemon:

   sudo usermod -aG docker $USER
   

6. Log out and log in again to the bootstrap node to apply the changes.

7. Verify that Docker is configured and works correctly. For example:

   docker run --rm alpine sh -c "apk add --no-cache curl; \
   curl https://binary.mirantis.com"
   

   The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

8. Proceed to Bootstrap a management cluster.

Bootstrap a management cluster¶

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your AWS-based Mirantis Container Cloud management cluster.

To bootstrap an AWS-based management cluster:

1. Log in to the bootstrap node running Ubuntu 18.04 that is configured as described in Prerequisites.

2. Download and run the Container Cloud bootstrap script:

   wget https://binary.mirantis.com/releases/get_container_cloud.sh
   
   chmod 0755 get_container_cloud.sh
   
   ./get_container_cloud.sh
   

3. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

4. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

5. Save the license file as mirantis.lic under the kaas-bootstrap directory.

6. Verify access to the target cloud endpoint from Docker. For example:

   docker run --rm alpine sh -c "apk add --no-cache curl; \
   curl https://ec2.amazonaws.com"
   

   The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

7. In templates/aws/machines.yaml.template, modify the spec:providerSpec:value section by substituting the ami:id parameter with the corresponding value for Ubuntu 18.04 from the required AWS region. For example:

    spec:
      providerSpec:
        value:
          apiVersion: aws.kaas.mirantis.com/v1alpha1
          kind: AWSMachineProviderSpec
          instanceType: c5d.2xlarge
          ami:
            id: ami-033a0960d9d83ead0
   

   Also, modify other parameters as required.

8. Optional. In templates/aws/cluster.yaml.template, modify the default AWS instance types and AMIs configuration for further creation of managed clusters:

   providerSpec:
       value:
         ...
         kaas:
           ...
           regional:
           - provider: aws
             helmReleases:
               - name: aws-credentials-controller
                 values:
                   config:
                     allowedInstanceTypes:
                       minVCPUs: 8
                       # in MiB
                       minMemory: 16384
                       # in GB
                       minStorage: 120
                       supportedArchitectures:
                       - "x86_64"
                       filters:
                       - name: instance-storage-info.disk.type
                         values:
                           - "ssd"
                     allowedAMIs:
                     -
                       - name: name
                         values:
                         - "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200729"
                       - name: owner-id
                         values:
                         - "099720109477"
   

   Also, modify other parameters as required.

9. Generate the AWS Access Key ID with Secret Access Key for the admin user and select the AWS default region name. For details, see AWS General Reference: Programmatic access.

10. Export the following parameters by adding the corresponding values for the AWS admin credentials created in the previous step:

    export KAAS_AWS_ENABLED=true
    export AWS_SECRET_ACCESS_KEY=XXXXXXX
    export AWS_ACCESS_KEY_ID=XXXXXXX
    export AWS_DEFAULT_REGION=us-east-2
    

11. Create the AWS CloudFormation template for IAM policy:

    ./kaas bootstrap aws policy
    

12. Generate the AWS Access Key ID with Secret Access Key for the bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user, that was created in the previous step, and select the AWS default region name.

13. Export the AWS bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user credentials that were created in the previous step:

    export KAAS_AWS_ENABLED=true
    export AWS_SECRET_ACCESS_KEY=XXXXXXX
    export AWS_ACCESS_KEY_ID=XXXXXXX
    export AWS_DEFAULT_REGION=us-east-2
    

14. Optional. Skip this step to use the default password password in the Container Cloud web UI.

    Configure the IAM parameters:

    1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

       ./bin/hash-generate -i 27500
       

       The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

       Example of system response:

       passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
       passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
       passwordHashAlgorithm: pbkdf2-sha256
       passwordHashIterations: 27500
       

       Run the tool several times to generate hashed passwords for every IAM role.

    2. Open templates/cluster.yaml.template for editing.

    3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

       • passwordSalt - base64-encoded randomly generated sequence of bytes.

       • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

15. Optional. Configure external identity provider for IAM.

16. Run the bootstrap script:

    ./bootstrap.sh all
    

17. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private SSH key openstack_tmp located in ~/.ssh/ for access to the management cluster nodes.

      Note

      The SSH key name openstack_tmp is the same for all cloud providers. This name will be changed in one of the following Container Cloud releases to avoid confusion with a cloud provider name and its related SSH key name.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

18. In case of deployment issues, collect and inspect the bootstrap and management cluster logs as described in Troubleshooting.

19. Optional. Deploy an additional regional cluster of a different provider type as described in Deploy an additional regional cluster.

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create an AWS-based managed cluster.

Workflow overview¶

Perform the following steps to install Mirantis Container Cloud on a VMWare vSphere-based environment:

1. Prepare a VMWare vSphere environment with the requirements described in Reference Architecture: VMWare vSphere-based cluster requirements.

2. Prepare the bootstrap node as described in Prerequisites.

3. Obtain the Mirantis license file to use during the bootstrap.

4. Set up the VMWare accounts for deployment as described in VMWare deployment users.

5. Prepare the OVF template for the management cluster nodes using OVF template requirements.

6. Create and configure the deployment configuration files that include the cluster and machines metadata.

7. Run the bootstrap script for the fully automated installation of the management cluster.

For more details, see Bootstrap a management cluster.

Prerequisites¶

Before bootstrapping a VMWare vSphere-based management cluster, complete the following prerequisite steps:

1. Verify that your planned cloud meets the reference hardware bill of material and software requirements as described in Reference Architecture: Requirements for a VMWare vSphere-based Container Cloud cluster.

2. Log in to any personal computer or VM running Ubuntu 18.04 that you will be using as the bootstrap node.

3. Install Docker version 18.09:

   sudo apt install docker.io
   

4. Grant your USER access to the Docker daemon:

   sudo usermod -aG docker $USER
   

5. Log off and log in again to the bootstrap node to apply the changes.

6. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

   docker run --rm alpine sh -c "apk add --no-cache curl; \
   curl https://binary.mirantis.com"
   

   The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

7. Prepare the VMWare deployment user setup and permissions.

8. Prepare the VMWare OVF template.

Prepare the VMWare deployment user setup and permissions¶

To deploy Mirantis Container Cloud on a VMWare vSphere-based environment, prepare the following VMWare accounts:

1. Create a read-only virt-who user on the vCenter Server. The virt-who user requires at least read-only access to all objects in the vCenter Data Center.

   The virt-who service on RHEL machines will be provided with the virt-who user credentials in order to properly manage RHEL subscriptions.

   For details on how to create the virt-who user, refer to the official RedHat Customer Portal documentation.

2. Add the following privileges to the cluster-api user:

   Note

   Container Cloud uses two separate vSphere accounts for the Cluster API related operations (create and delete VMs) and for storage operations (dynamic PVC provision). But also it can be one user that has both privileges sets.

   Privilege	Permission
   Content Library	Download files Read storage Sync library item
   Datastore	Allocate space Browse datastore Update virtual machine files Update virtual machine metadata
   Folder	Create folder Rename folder
   Global	Cancel task
   Host	Local operations: Create virtual machine Delete virtual machine Reconfigure virtual machine
   Network	Assign network
   Network	Assign virtual machine to resource pool
   Scheduled task	Create tasks Modify task Remove task Run task
   Sessions	Validate session View and stop sessions
   Storage views	View
   Tasks	Create task Update task

   Virtual machine permissions¶

   Privilege	Permission
   Change configuration	Acquire disk lease Add existing disk Add new disk Add or remove device Advanced configuration Change CPU count Change Memory Change Settings Change Swapfile placement Change resource Configure Host USB device Configure Raw device Configure managedBy Display connection settings Extend virtual disk Modify device settings Query Fault Tolerance compatibility Query unowned files Reload from path Remove disk Rename Reset guest information Set annotation Toggle disk change tracking Toggle fork parent Upgrade virtual machine compatibility
   Edit inventory	Create from existing Create new Move Register Remove Unregister
   Interaction	Connect devices Console interaction Power off Power on Reset Suspend
   Provisioning	Allow disk access Allow file access Allow read-only disk access Allow virtual machine download Allow virtual machine files upload Clone template Clone virtual machine Create template from virtual machine Customize guest Deploy template Mark as template Mark as virtual machine Modify customization specification Promote disks Read customization specifications
   Snapshot management	Create snapshot Remove snapshot Rename snapshot Revert to snapshot
   vSphere replication	Monitor replication

3. Add the following privileges to the storage user:

   Privilege	Permission
   Cns	Searchable
   Content Library	View configuration settings
   Datastore	Allocate space Browse datastore Configure datastore Low level file operations Remove file Update virtual machine files Update virtual machine metadata
   Folder	Create folder Delete folder Move folder Rename folder
   Host	Configuration: Storage partition configuration Local operations: Create virtual machine Delete virtual machine Reconfigure virtual machine
   ImageBuilder	Depot: Create Delete Profile: Create Delete Export
   Host profile	View
   Resource	Assign vApp to resource pool Assign virtual machine to resource pool Migrate powered off virtual machine Migrate powered on virtual machine Modify resource pool Query vMotion
   Scheduled task	Create tasks Modify task Run task
   Sessions	Validate session View and stop sessions
   Datastore cluster	Configure a datastore cluster
   Profile-driven storage	Profile-driven storage view
   Storage views	Configure service View

   Virtual machine permissions¶

   Privilege	Permission
   Change configuration	Acquire disk lease Add existing disk Add new disk Add or remove device Advanced configuration Change CPU count Change Memory Change Settings Change Swapfile placement Change resource Configure Host USB device Configure Raw device Configure managedBy Display connection settings Extend virtual disk Modify device settings Query Fault Tolerance compatibility Query unowned files Reload from path Remove disk Rename Reset guest information Set annotation Toggle disk change tracking Toggle fork parent Upgrade virtual machine compatibility
   Provisioning	Allow disk access Allow file access Allow read-only disk access Allow virtual machine download Allow virtual machine files upload Clone template Clone virtual machine Create template from virtual machine Customize guest Deploy template Mark as template Mark as virtual machine Modify customization specification Promote disks Read customization specifications
   Snapshot management	Create snapshot Remove snapshot Rename snapshot Revert to snapshot

Now, proceed to Prepare the VMWare OVF template.

Prepare the VMWare OVF template¶

To deploy Mirantis Container Cloud on a VMWare vSphere-based environment, the VMWare OVF template for cluster machines must be prepared according to the following requirements:

1. The VMware Tools package is installed.

2. The cloud-init utility is installed and configured with the specific VMwareGuestInfo datasource.

3. The virt-who service is enabled and configured to connect to the VMware vCenter Server to properly apply the RHEL subscriptions on the nodes.

The following procedure describes how to meet the requirements above.

To prepare the VMWare OVF template:

1. Run a virtual machine on the VMware Datacenter from the official RHEL 7.8 server image. Specify the amount of resources that will be used in the Container Cloud setup. A minimal resources configuration must match the requirements for a VMWare vSphere-based Container Cloud cluster.

2. Select minimal setup in the VM installation configuration. Create a user with root or sudo permissions to access the machine.

3. Log in to the VM when it starts.

4. Attach your RHEL license for Virtual Datacenters to the VM:

   subscription-manager register
   # automatic subscription selection:
   subscription-manager attach --auto
   # or specify pool id:
   subscription-manager attach --pool=<POOL_ID>
   # verify subscription status
   subscription-manager status
   

5. Select from the following options:

   • Prepare the operating system automatically:

     1. Download the automation script:

        curl https://gerrit.mcp.mirantis.com/plugins/gitiles/kubernetes/vmware-guestinfo/+/refs/tags/  v1.1.1/install.sh?format=TEXT | \
        base64 -d > install.sh
        chmod +x install.sh
        

     2. Export the VMware vCenter Server credentials of the read-only user. For example:

        export VC_SERVER='vcenter1.example.com'
        export VC_USER='domain\vmware_read_only_username'
        export VC_PASSWORD='password!23'
        export VC_OWNER='1234567'
        # optional parameter:
        export VC_HYPERVISOR_ID=hostname
        export VC_FILTER_HOSTS="esx1.example.com, esx2.example.com"
        export VCENTER_CONFIG_PATH="/etc/virt-who.d/vcenter.conf"
        

     3. Run the installation script:

        ./install.sh
        

   • Prepare the operating system manually:

     1. Install open-vm-tools:

        yum install open-vm-tools -y
        

     2. Install and configure cloud-init:

        1. Download the VMwareGuestInfo datasource files:

           curl https://gerrit.mcp.mirantis.com/plugins/gitiles/kubernetes/vmware-guestinfo/+/refs/tags/v1.1.1/DataSourceVMwareGuestInfo.py?format=TEXT | \
           base64 -d > DataSourceVMwareGuestInfo.py
           curl https://gerrit.mcp.mirantis.com/plugins/gitiles/kubernetes/vmware-guestinfo/+/refs/tags/v1.1.1/99-DataSourceVMwareGuestInfo.cfg?format=TEXT | \
           base64 -d > 99-DataSourceVMwareGuestInfo.cfg
           

        2. Add 99-DataSourceVMwareGuestInfo.cfg to /etc/cloud/cloud.cfg.d/.

        3. Depending on the Python version on the VM operating system, add DataSourceVMwareGuestInfo.py to the cloud-init sources folder.

        4. Obtain the cloud-init folder on RHEL:

           yum install cloud-init -y
           python -c 'import os; from cloudinit import sources; print(os.path.dirname(sources.__file__));'
           

     3. Prepare the virt-who user configuration:

        Note

        For details about the virt-who user creation, see Prepare the VMWare deployment user setup and permissions.

        1. Install virt-who:

           yum install virt-who -y
           cp /etc/virt-who.d/template.conf /etc/virt-who.d/vcenter.conf
           

        2. Set up the file content using the following example:

           [vcenter]
           type=esx
           server=vcenter1.example.com
           username=domain\vmware_read_only_username
           encrypted_password=bd257f93d@482B76e6390cc54aec1a4d
           owner=1234567
           hypervisor_id=hostname
           filter_hosts=esx1.example.com, esx2.example.com
           

           virt-who configuration parameters¶

           Parameter	Description
           [vcenter]	Name of VMware datacenter.
           type=esx	Specifies the connection of the defined virt-who user to the VMware vCenter Server.
           server	The FQDN of the vCenter Server.
           username	The virt-who user name on the vCenter Server with the read-only access.
           encrypted_password	The virt-who password encrypted by the virt-who-password utility using the virt-who-password -p <password> command.
           owner	The organization that the hypervisors belong to.
           hypervisor_id	Specifies how to identify the hypervisors. Use a host name to provide meaningful host names to the Subscription Management. Alternatively, use uuid or hwuuid to avoid duplication in case of hypervisor renaming.
           filter_hosts	List of hypervisors that never run RHEL VMs. Such hypervisors do not have to be reported by virt-who.

6. Remove the RHEL subscription from the node.

   subscription-manager remove --all
   subscription-manager unregister
   subscription-manager clean
   

7. Shut down the VM.

8. Create an OVF template from the VM.

Now, proceed to Bootstrap a management cluster.

Bootstrap a management cluster¶

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your VMWare vSphere-based Mirantis Container Cloud management cluster.

To bootstrap a VMWare vSphere-based management cluster:

1. Log in to the bootstrap node running Ubuntu 18.04 that is configured as described in Prerequisites.

2. Download and run the Container Cloud bootstrap script:

   wget https://binary.mirantis.com/releases/get_container_cloud.sh
   chmod 0755 get_container_cloud.sh
   ./get_container_cloud.sh
   

3. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script.

4. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

5. Save the license file as mirantis.lic under the kaas-bootstrap directory.

6. In templates/vsphere/rhellicenses.yaml.template, set the user name and password of your RedHat Customer Portal account associated with your RHEL license for Virtual Datacenters. Optionally, specify the subscription allocation pools to use for the RHEL subscriptions activation. If you leave the pool field empty, subscription-manager will automatically select the licenses for machines.

7. In templates/vsphere/machines.yaml.template, modify the spec:providerSpec:value section using the following configuration file extract as an example:

   spec:
     providerSpec:
       value:
         apiVersion: vsphere.cluster.k8s.io/v1alpha1
         kind: VsphereMachineProviderSpec
         sshUserName: SSH_USER_NAME
         rhelLicense: kaas-mgmt-rhel-license
         datacenter: SET_VSPHERE_DATACENTER_PATH
         network:
           devices:
           - networkName: SET_VSPHERE_NETWORK_PATH
             dhcp4: true
             dhcp6: false
         template: SET_VSPHERE_TEMPLATE_PATH
   

   Replace the values in caps lock with the corresponding values of your environment. The default SSH user name is cloud-user. Also, modify other parameters as required.

8. Modify the templates/vsphere/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

   Note

   The passwordSalt and passwordHash values for the IAM roles are automatically re-generated during the IAM configuration described in the next step.

9. Modify vsphere-config.yaml.template:

   vSphere configuration data¶

   Parameter	Description
   SET_VSPHERE_SERVER_IP	IP address of the VMware vCenter Server.
   SET_VSPHERE_CAPI_PROVIDER_USERNAME	vSphere Cluster API provider user name. For details, see Prepare the VMWare deployment user setup and permissions.
   SET_VSPHERE_CAPI_PROVIDER_PASSWORD	vSphere Cluster API provider password.
   SET_VSPHERE_CLOUD_PROVIDER_USERNAME	VMware deployment user name. For details, see Prepare the VMWare deployment user setup and permissions.
   SET_VSPHERE_CLOUD_PROVIDER_PASSWORD	VMware deployment user password.
   SET_VSPHERE_DATACENTER_NAME	VMware datacenter name.
   SET_VSPHERE_DATACENTER_PATH	VMware datacenter path.
   SET_VSPHERE_DATASTORE_NAME	VMware datastore name.
   SET_VSPHERE_NETWORK_PATH	Path to network for cluster machines.
   SET_VSPHERE_RESOURCE_POOL_PATH	Path to a resource pool in which VMs will be created.
   SET_VSPHERE_MACHINES_FOLDER	Path to a folder where the cluster machines metadata will be stored.
   SET_VSPHERE_SERVER_INSECURE	Flag that controls validation of the vSphere Server certificate.
   SET_VSPHERE_TEMPLATE_PATH	Path to the prepared OVF template for cluster machines.

10. Optional. Skip this step to use the default password password in the Container Cloud web UI.

    Configure the IAM parameters:

    1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

       ./bin/hash-generate -i 27500
       

       The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

       Example of system response:

       passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
       passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
       passwordHashAlgorithm: pbkdf2-sha256
       passwordHashIterations: 27500
       

       Run the tool several times to generate hashed passwords for every IAM role.

    2. Open templates/cluster.yaml.template for editing.

    3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

       • passwordSalt - base64-encoded randomly generated sequence of bytes.

       • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

11. Optional. Configure external identity provider for IAM.

12. Run the bootstrap script:

    ./bootstrap.sh all
    

13. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private SSH key openstack_tmp located in ~/.ssh/ for access to the management cluster nodes.

      Note

      The SSH key name openstack_tmp is the same for all cloud providers. This name will be changed in one of the following Container Cloud releases to avoid confusion with a cloud provider name and its related SSH key name.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create a VMWare vSphere-based managed cluster.

Deploy an AWS-based regional cluster¶

If you want to deploy AWS-based managed clusters of different configurations, deploy an additional regional cluster with specific settings that differ from the AWS-based management cluster configuration.

To deploy an AWS-based regional cluster:

1. Log in to the node where you bootstrapped a management cluster.

2. Prepare the AWS configuration for the new regional cluster:

   1. Verify access to the target cloud endpoint from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://ec2.amazonaws.com"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

   2. Change the directory to the kaas-bootstrap folder.

   3. In templates/aws/machines.yaml.template, modify the spec:providerSpec:value section by substituting the ami:id parameter with the corresponding value for Ubuntu 18.04 from the required AWS region. For example:

       spec:
         providerSpec:
           value:
             apiVersion: aws.kaas.mirantis.com/v1alpha1
             kind: AWSMachineProviderSpec
             instanceType: c5d.2xlarge
             ami:
               id: ami-033a0960d9d83ead0
      

      Also, modify other parameters as required.

   4. Optional. In templates/aws/cluster.yaml.template, modify the default configuration of the AWS instance types and AMI IDs for further creation of managed clusters:

      providerSpec:
          value:
            ...
            kaas:
              ...
              regional:
              - provider: aws
                helmReleases:
                  - name: aws-credentials-controller
                    values:
                      config:
                        allowedInstanceTypes:
                          minVCPUs: 8
                          # in MiB
                          minMemory: 16384
                          # in GB
                          minStorage: 120
                          supportedArchitectures:
                          - "x86_64"
                          filters:
                          - name: instance-storage-info.disk.type
                            values:
                              - "ssd"
                        allowedAMIs:
                        -
                          - name: name
                            values:
                            - "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200729"
                          - name: owner-id
                            values:
                            - "099720109477"
      

      Also, modify other parameters as required.

   5. Generate the AWS Access Key ID with Secret Access Key for the bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user, that was created in the previous step, and select the AWS default region name.

   6. Export the AWS bootstrapper.cluster-api-provider-aws.kaas.mirantis.com user credentials that were created in the previous step:

      export KAAS_AWS_ENABLED=true
      export AWS_SECRET_ACCESS_KEY=XXXXXXX
      export AWS_ACCESS_KEY_ID=XXXXXXX
      export AWS_DEFAULT_REGION=us-east-2
      

3. Export the following parameters:

   export KUBECONFIG=<pathToMgmtClusterKubeconfig>
   export REGIONAL_CLUSTER_NAME=<newRegionalClusterName>
   export REGION=<NewRegionName>
   

   Substitute the parameters enclosed in angle brackets with the corresponding values of your cluster.

4. Run the regional cluster bootstrap script:

   ./bootstrap.sh deploy_regional
   

   Note

   When the bootstrap is complete, obtain and save in a secure location the kubeconfig-<regionalClusterName> file located in the same directory as the bootstrap script. This file contains the admin credentials for the regional cluster.

   The workflow of the regional cluster bootstrap script¶

   #	Description
   1	Prepare the bootstrap cluster for the new regional cluster.
   2	Load the updated Container Cloud CRDs for Credentials, Cluster, and Machines with information about the new regional cluster to the management cluster.
   3	Connect to each machine of the management cluster through SSH.
   4	Wait for the Machines and Cluster objects of the new regional cluster to be ready on the management cluster.
   5	Load the following objects to the new regional cluster: Secret with the management cluster kubeconfig and ClusterRole for the Container Cloud provider.
   6	Forward the bootstrap cluster endpoint to helm-controller.
   7	Wait for all CRDs to be available and verify the objects created using these CRDs.
   8	Pivot the cluster API stack to the regional cluster.
   9	Switch the LCM agent from the bootstrap cluster to the regional one.
   10	Wait for the Container Cloud components to start on the regional cluster.

Now, you can proceed with deploying the managed clusters of supported provider types as described in Create and operate a managed cluster.

Deploy an OpenStack-based regional cluster¶

You can deploy an additional regional OpenStack-based cluster on top of the AWS, bare metal, or OpenStack management cluster to create managed clusters of several provider types if required.

To deploy an OpenStack-based regional cluster:

1. Log in to the node where you bootstrapped a management cluster.

2. Prepare the OpenStack configuration for a new regional cluster:

   1. Log in to the OpenStack Horizon.

   2. In the Project section, select API Access.

   3. In the right-side drop-down menu Download OpenStack RC File, select OpenStack clouds.yaml File.

   4. Add the downloaded clouds.yaml file to the directory with the bootstrap.sh script.

   5. In clouds.yaml, add the password field with your OpenStack password under the clouds/openstack/auth section.

      Example:

      clouds:
        openstack:
          auth:
            auth_url: https://auth.openstack.example.com:5000/v3
            username: your_username
            password: your_secret_password
            project_id: your_project_id
            user_domain_name: your_ldap_password
          region_name: RegionOne
          interface: public
          identity_api_version: 3
      

   6. Verify access to the target cloud endpoint from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
      curl https://auth.openstack.example.com:5000/v3"
      

      The system output must contain no error records. In case of issues, follow the steps provided in Troubleshooting.

   7. Change the directory to the kaas-bootstrap folder.

   8. In templates/machines.yaml.template, modify the spec:providerSpec:value sections for set: master and set: node by substituting the flavor and image parameters with the corresponding values of your OpenStack cluster. For example:

      spec:
        providerSpec:
          value:
            apiVersion: "openstackproviderconfig.k8s.io/v1alpha1"
            kind: "OpenstackMachineProviderSpec"
            flavor: kaas.minimal
            image: bionic-server-cloudimg-amd64-20190612
      

      Also, modify other parameters as required.

   9. Modify the templates/cluster.yaml.template parameters to fit your deployment. For example, add the corresponding values for cidrBlocks in the spec::clusterNetwork::services section.

      Note

      The passwordSalt and passwordHash values for the IAM roles are automatically re-generated during the IAM configuration described in the next step.

3. Clean up the environment configuration:

   1. If you are deploying the regional cluster on top of a baremetal-based management cluster, unset the following parameters:

      unset KAAS_BM_ENABLED KAAS_BM_FULL_PREFLIGHT KAAS_BM_PXE_IP \
            KAAS_BM_PXE_MASK KAAS_BM_PXE_BRIDGE KAAS_BM_BM_DHCP_RANGE \
            TEMPLATES_DIR
      

   2. If you are deploying the regional cluster on top of an AWS-based management cluster, unset the KAAS_AWS_ENABLED parameter:

      unset KAAS_AWS_ENABLED
      

4. Export the following parameters:

   export KUBECONFIG=<pathToMgmtClusterKubeconfig>
   export REGIONAL_CLUSTER_NAME=<newRegionalClusterName>
   export REGION=<NewRegionName>
   

   Substitute the parameters enclosed in angle brackets with the corresponding values of your cluster.

5. Run the regional cluster bootstrap script:

   ./bootstrap.sh deploy_regional
   

   Note

   When the bootstrap is complete, obtain and save in a secure location the kubeconfig-<regionalClusterName> file located in the same directory as the bootstrap script. This file contains the admin credentials for the regional cluster.

   The workflow of the regional cluster bootstrap script¶

   #	Description
   1	Prepare the bootstrap cluster for the new regional cluster.
   2	Load the updated Container Cloud CRDs for Credentials, Cluster, and Machines with information about the new regional cluster to the management cluster.
   3	Connect to each machine of the management cluster through SSH.
   4	Wait for the Machines and Cluster objects of the new regional cluster to be ready on the management cluster.
   5	Load the following objects to the new regional cluster: Secret with the management cluster kubeconfig and ClusterRole for the Container Cloud provider.
   6	Forward the bootstrap cluster endpoint to helm-controller.
   7	Wait for all CRDs to be available and verify the objects created using these CRDs.
   8	Pivot the cluster API stack to the regional cluster.
   9	Switch the LCM agent from the bootstrap cluster to the regional one.
   10	Wait for the Container Cloud components to start on the regional cluster.

Now, you can proceed with deploying the managed clusters of supported provider types as described in Create and operate a managed cluster.

Collect the bootstrap logs¶

If the bootstrap script fails during the deployment process, collect and inspect the bootstrap and management cluster logs.

To collect the bootstrap logs:

1. Log in to your local machine where the bootstrap script was executed.

2. Run the following command:

   ./bootstrap.sh collect_logs
   

   The logs are collected in the directory where the bootstrap script is located.

Starting from the Container Cloud release 2.2.0, the logs structure is as follows:

• <output_dir>/<cluster_name>/events.log - human-readable table that contains information about the cluster events

• <output_dir>/<cluster_name>/system - system logs

• <output_dir>/<cluster_name>/objects/cluster - logs of the non-namespaced Kubernetes objects

• <output_dir>/<cluster_name>/objects/namespaced - logs of the namespaced Kubernetes objects

• <output_dir>/<cluster_name>/objects/namespaced/<namespace>/core/pods - pods logs from a specified Kubernetes namespace

Depending on the type of issue found in logs, apply the corresponding fixes. For example, if you detect the LoadBalancer ERROR state errors during the bootstrap of an OpenStack-based management cluster, contact your system administrator to fix the issue. To troubleshoot other issues, refer to the corresponding section in Troubleshooting.

DNS settings¶

If you have issues related to the DNS settings, the following error message may occur:

curl: (6) Could not resolve host

The issue may occur if a VPN is used to connect to the cloud or a local DNS forwarder is set up.

The workaround is to change the default DNS settings for Docker:

1. Log in to your local machine.

2. Identify your internal or corporate DNS server address:

   systemd-resolve --status
   

3. Create or edit /etc/docker/daemon.json by specifying your DNS address:

   {
     "dns": ["<YOUR_DNS_ADDRESS>"]
   }
   

4. Restart the Docker daemon:

   sudo systemctl restart docker
   

Default network address¶

If you have issues related to the default network address configuration, cURL either hangs or the following error occurs:

curl: (7) Failed to connect to xxx.xxx.xxx.xxx port xxxx: Host is unreachable

The issue may occur because the default Docker network address 172.17.0.0/16 overlaps with your cloud address or other addresses of the network configuration.

Workaround:

1. Log in to your local machine.

2. Verify routing to the IP addresses of the target cloud endpoints:

   1. Obtain the IP address of your target cloud. For example:

      nslookup auth.openstack.example.com
      

      Example of system response:

      Name:   auth.openstack.example.com
      Address: 172.17.246.119
      

   2. Verify that this IP address is not routed through docker0 but through any other interface, for example, ens3:

      ip r get 172.17.246.119
      

      Example of the system response if the routing is configured correctly:

      172.17.246.119 via 172.18.194.1 dev ens3 src 172.18.1.1 uid 1000
        cache
      

      Example of the system response if the routing is configured incorrectly:

      172.17.246.119 via 172.18.194.1 dev docker0 src 172.18.1.1 uid 1000
        cache
      

3. If the routing is incorrect, change the IP address of the default Docker bridge:

   1. Create or edit /etc/docker/daemon.json by adding the "bip" option:

      {
        "bip": "192.168.91.1/24"
      }
      

   2. Restart the Docker daemon:

      sudo systemctl restart docker
      

TLS handshake timeout¶

If you execute the bootstrap.sh script from an OpenStack VM that is running on the OpenStack environment used for bootstrapping the management cluster, the following error messages may occur that can be related to the MTU settings discrepancy:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to server:port

Failed to check if machine "<machine_name>" exists:
failed to create provider client ... TLS handshake timeout

To identify whether the issue is MTU-related:

1. Log in to the OpenStack VM in question.

2. Compare the MTU outputs for the docker0 and ens3 interfaces:

   ip addr
   

   Example of system response:

   3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500...
   ...
   2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450...
   

   If the MTU output values differ for docker0 and ens3, proceed with the workaround below. Otherwise, inspect the logs further to identify the root cause of the error messages.

Workaround:

1. In your OpenStack environment used for Mirantis Container Cloud, log in to any machine with CLI access to OpenStack. For example, you can create a new Ubuntu VM (separate from the bootstrap VM) and install the python-openstackclient package on it.

2. Change the vXLAN MTU size for the VM to the required value depending on your network infrastructure and considering your physical network configuration, such as Jumbo frames, and so on.

   openstack network set --mtu <YOUR_MTU_SIZE> <network-name>
   

3. Stop and start the VM in Nova.

4. Log in to the bootstrap VM dedicated for the management cluster.

5. Re-execute the bootstrap.sh script.

Configure LDAP for IAM¶

If you integrate LDAP for IAM to Mirantis Container Cloud, add the required LDAP configuration to cluster.yaml.template during the bootstrap of the management cluster.

authType: "simple"
bindCredential: ""
bindDn: ""

To configure LDAP for IAM:

1. Select from the following options:

   • For a baremetal-based management cluster, open the templates/bm/cluster.yaml.template file for editing.

   • For an OpenStack management cluster, open the templates/cluster.yaml.template file for editing.

   • For an AWS-based management cluster, open the templates/aws/cluster.yaml.template file for editing.

2. Configure the keycloak:userFederation:providers: and keycloak:userFederation:mappers: sections as required:

   Note

   Verify that the userFederation section is located on the same level as the initUsers section.

   spec:
     providerSpec:
       value:
         kaas:
           management:
             helmReleases:
             - name: iam
               values:
                 keycloak:
                   userFederation:
                     providers:
                       - displayName: "<LDAP_NAME>"
                         providerName: "ldap"
                         priority: 1
                         fullSyncPeriod: -1
                         changedSyncPeriod: -1
                         config:
                           pagination: "true"
                           debug: "false"
                           searchScope: "1"
                           connectionPooling: "true"
                           usersDn: "<DN>" # "ou=People, o=<ORGANIZATION>, dc=<DOMAIN_COMPONENT>"
                           userObjectClasses: "inetOrgPerson,organizationalPerson"
                           usernameLDAPAttribute: "uid"
                           rdnLDAPAttribute: "uid"
                           vendor: "ad"
                           editMode: "READ_ONLY"
                           uuidLDAPAttribute: "uid"
                           connectionUrl: "ldap://<LDAP_DNS>"
                           syncRegistrations: "false"
                           authType: "simple"
                           bindCredential: ""
                           bindDn: ""
                     mappers:
                       - name: "username"
                         federationMapperType: "user-attribute-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.attribute: "uid"
                           user.model.attribute: "username"
                           is.mandatory.in.ldap: "true"
                           read.only: "true"
                           always.read.value.from.ldap: "false"
                       - name: "full name"
                         federationMapperType: "full-name-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.full.name.attribute: "cn"
                           read.only: "true"
                           write.only: "false"
                       - name: "last name"
                         federationMapperType: "user-attribute-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.attribute: "sn"
                           user.model.attribute: "lastName"
                           is.mandatory.in.ldap: "true"
                           read.only: "true"
                           always.read.value.from.ldap: "true"
                       - name: "email"
                         federationMapperType: "user-attribute-ldap-mapper"
                         federationProviderDisplayName: "<LDAP_NAME>"
                         config:
                           ldap.attribute: "mail"
                           user.model.attribute: "email"
                           is.mandatory.in.ldap: "false"
                           read.only: "true"
                           always.read.value.from.ldap: "true"
   

Now, return to the bootstrap instruction depending on the provider type of your management cluster.

Configure Google OAuth IdP for IAM¶

If you integrate Google OAuth external identity provider for IAM to Mirantis Container Cloud, create the authorization credentials for IAM in your Google OAuth account and configure cluster.yaml.template during the bootstrap of the management cluster.

To configure Google OAuth IdP for IAM:

1. Create Google OAuth credentials for IAM:

   1. Log in to your https://console.developers.google.com.

   2. Navigate to Credentials.

   3. In the APIs Credentials menu, select OAuth client ID.

   4. In the window that opens:

      1. In the Application type menu, select Web application.

      2. In the Authorized redirect URIs field, type in <keycloak-url>/auth/realms/iam/broker/google/endpoint, where <keycloak-url> is the corresponding DNS address.

      3. Press Enter to add the URI.

      4. Click Create.

      A page with your client ID and client secret opens. Save these credentials for further usage.

2. Log in to the bootstrap node.

3. Select from the following options:

   • For a baremetal-based management cluster, open the templates/bm/cluster.yaml.template file for editing.

   • For an OpenStack management cluster, open the templates/cluster.yaml.template file for editing.

   • For an AWS-based management cluster, open the templates/aws/cluster.yaml.template file for editing.

4. In the keycloak:externalIdP: section, add the following snippet with your credentials created in previous steps:

   keycloak:
     externalIdP:
       google:
         enabled: true
         config:
           clientId: <Google_OAuth_client_ID>
           clientSecret: <Google_OAuth_client_secret>
   

Now, return to the bootstrap instruction depending on the provider type of your management cluster.