Bootstrap a management cluster

After you complete the prerequisite steps described in Prerequisites, proceed with bootstrapping your OpenStack-based Mirantis Container Cloud management cluster.

To bootstrap an OpenStack-based management cluster:

  1. Log in to the bootstrap node running Ubuntu 20.04 that is configured as described in Prerequisites.

  2. Prepare the bootstrap script:

    1. Download and run the Container Cloud bootstrap script:

      sudo apt-get update
sudo apt-get install wget
wget https://binary.mirantis.com/releases/get_container_cloud.sh
chmod 0755 get_container_cloud.sh
./get_container_cloud.sh

    2. Change the directory to the kaas-bootstrap folder created by the script.

  3. Obtain your license file that will be required during the bootstrap:

    1. Select from the following options:

      • Open the email from support@mirantis.com with the subject Mirantis Container Cloud License File or Mirantis OpenStack License File

      • In the Mirantis CloudCare Portal, open the Account or Cloud page

    2. Download the License File and save it as mirantis.lic under the kaas-bootstrap directory on the bootstrap node.

    3. Verify that mirantis.lic contains the previously downloaded Container Cloud license by decoding the license JWT token, for example, using jwt.io.

      Example of a valid decoded Container Cloud license data with the mandatory license field:

      {
    "exp": 1652304773,
    "iat": 1636669973,
    "sub": "demo",
    "license": {
        "dev": false,
        "limits": {
            "clusters": 10,
            "workers_per_cluster": 10
        },
        "openstack": null
    }
}

      Warning

      The MKE license does not apply to mirantis.lic. For details about MKE license, see MKE documentation.

  4. Prepare the OpenStack configuration for a new cluster:

    1. Log in to the OpenStack Horizon.

    2. In the Project section, select API Access.

    3. In the right-side drop-down menu Download OpenStack RC File, select OpenStack clouds.yaml File.

    4. Save the downloaded clouds.yaml file in the kaas-bootstrap folder created by the get_container_cloud.sh script.

    5. In clouds.yaml, add the password field with your OpenStack password under the clouds/openstack/auth section.

      Example:

      clouds:
  openstack:
    auth:
      auth_url: https://auth.openstack.example.com/v3
      username: your_username
      password: your_secret_password
      project_id: your_project_id
      user_domain_name: your_user_domain_name
    region_name: RegionOne
    interface: public
    identity_api_version: 3

    6. If you deploy Container Cloud on top of MOSK Victoria with Tungsten Fabric and use the default security group for newly created load balancers, add the following rules for the Kubernetes API server endpoint, Container Cloud application endpoint, and for the MKE web UI and API using the OpenStack CLI:

      • direction='ingress'

      • ethertype='IPv4'

      • protocol='tcp'

      • remote_ip_prefix='0.0.0.0/0'

      • port_range_max and port_range_min:

        • '443' for Kubernetes API and Container Cloud application endpoints

        • '6443' for MKE web UI and API

    7. Verify access to the target cloud endpoint from Docker. For example:

      docker run --rm alpine sh -c "apk add --no-cache curl; \
curl https://auth.openstack.example.com/v3"

      The system output must contain no error records.

    In case of issues, follow the steps provided in Troubleshooting.

  5. Configure the cluster and machines metadata:

    1. Adjust the templates/cluster.yaml.template parameters to suit your deployment:

      1. In the spec::providerSpec::value section, add the mandatory ExternalNetworkID parameter that is the ID of an external OpenStack network. It is required to have public Internet access to virtual machines.

      2. In the spec::clusterNetwork::services section, add the corresponding values for cidrBlocks.

      3. Configure other parameters as required.

    2. In templates/machines.yaml.template, modify the spec:providerSpec:value section for 3 control plane nodes marked with the cluster.sigs.k8s.io/control-plane label by substituting the flavor and image parameters with the corresponding values of the control plane nodes in the related OpenStack cluster. For example:

      spec: &cp_spec
  providerSpec:
    value:
      apiVersion: "openstackproviderconfig.k8s.io/v1alpha1"
      kind: "OpenstackMachineProviderSpec"
      flavor: kaas.minimal
      image: bionic-server-cloudimg-amd64-20190612

      Note

      The flavor parameter value provided in the example above is cloud-specific and must meet the Container Cloud requirements.

      Also, modify other parameters as required.

  6. Available since Container Cloud 2.24.0. Optional. Technology Preview. Enable custom host names for cluster machines. When enabled, any machine host name in a particular region matches the related Machine object name. For example, instead of the default kaas-node-<UID>, a machine host name will be master-0. The custom naming format is more convenient and easier to operate with.

    To enable the feature on the management and its future managed clusters:

    1. In templates/cluster.yaml.template, find the spec.providerSpec.value.kaas.regional section of the required region.

    2. In this section, find the required provider name under helmReleases.

    3. Under values.config, add customHostnamesEnabled: true.

      For example, for the bare metal provider in region-one:

      regional:
 - helmReleases:
   - name: baremetal-provider
     values:
       config:
         allInOneAllowed: false
         customHostnamesEnabled: true
         internalLoadBalancers: false
   provider: baremetal-provider

    Add the following environment variable:

    export CUSTOM_HOSTNAMES=true

  7. Optional. Available as TechPreview. To boot cluster machines from a block storage volume, define the following parameter in the spec:providerSpec section of templates/machines.yaml.template:

    bootFromVolume:
  enabled: true
  volumeSize: 120

    Note

    The minimal storage requirement is 120 GB per node. For details, see Requirements for an OpenStack-based cluster.

    To boot the Bastion node from a volume, add the same parameter to templates/cluster.yaml.template in the spec:providerSpec section for Bastion. The default amount of storage 80 is enough.

  8. Optional. Available since Container Cloud 2.24.0 as Technology Preview. Create all load balancers of the cluster with a specific Octavia flavor by defining the following parameter in the spec:providerSpec section of templates/cluster.yaml.template:

    serviceAnnotations:
  loadbalancer.openstack.org/flavor-id: <octaviaFlavorID>

    For details, see OpenStack documentation: Octavia Flavors.

    Note

    This feature is not supported by OpenStack Queens.

  9. Optional. Configure backups for the MariaDB database as described in Configure periodic backups of MariaDB for the OpenStack provider.

  10. Configure NTP server.

    Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool (*.ubuntu.pool.ntp.org) are accessible from the node where the management cluster is being provisioned. Otherwise, configure the regional NTP server parameters as described below.

    Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the management of chrony configuration by Container Cloud to use your own system for chrony management. Otherwise, configure the regional NTP server parameters as described below.

    NTP configuration

    Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

    In templates/cluster.yaml.template, add the ntp:servers section with the list of required server names:

    spec:
  ...
  providerSpec:
    value:
      kaas:
      ...
      ntpEnabled: true
        regional:
          - helmReleases:
            - name: <providerName>-provider
              values:
                config:
                  lcm:
                    ...
                    ntp:
                      servers:
                      - 0.pool.ntp.org
                      ...
            provider: <providerName>
            ...

    To disable NTP:

    spec:
  ...
  providerSpec:
    value:
      ...
      ntpEnabled: false
      ...

  11. Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the management and regional cluster using proxy:

    • HTTP_PROXY

    • HTTPS_PROXY

    • NO_PROXY

    • PROXY_CA_CERTIFICATE_PATH

    Example snippet:

    export HTTP_PROXY=http://proxy.example.com:3128
export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
export NO_PROXY=172.18.10.0,registry.internal.lan
export PROXY_CA_CERTIFICATE_PATH="/home/ubuntu/.mitmproxy/mitmproxy-ca-cert.cer"

    The following formats of variables are accepted:

    Proxy configuration data

    Variable

    Format
    HTTP_PROXY
    HTTPS_PROXY

    • http://proxy.example.com:port - for anonymous access.

    • http://user:password@proxy.example.com:port - for restricted access.

    NO_PROXY

    Comma-separated list of IP addresses or domain names.

    PROXY_CA_CERTIFICATE_PATH

    Optional. Absolute path to the proxy CA certificate for man-in-the-middle (MITM) proxies. Must be placed on the bootstrap node to be trusted. For details, see Install a CA certificate for a MITM proxy on a bootstrap node.

    Warning

    If you require Internet access to go through a MITM proxy, ensure that the proxy has streaming enabled as described in Enable streaming for MITM.

    Note

    For MOSK-based deployments, the parameter is generally available since MOSK 22.4.

    For implementation details, see Proxy and cache support.

    For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Requirements for an OpenStack-based cluster.

  12. Available since Container Cloud 2.24.0 and 2.24.2 for MOSK 23.2. Optional. Technology Preview. Enable the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.

    Configuration for auditd

    In templates/cluster.yaml.template, add the auditd parameters:

    spec:
  providerSpec:
    value:
      audit:
        auditd:
          enabled: <bool>
          enabledAtBoot: <bool>
          backlogLimit: <int>
          maxLogFile: <int>
          maxLogFileAction: <string>
          maxLogFileKeep: <int>
          mayHaltSystem: <bool>
          presetRules: <string>
          customRules: <string>
          customRulesX32: <text>
          customRulesX64: <text>

    Configuration parameters for auditd:

    enabled

    Boolean, default - false. Enables the auditd role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.

    enabledAtBoot

    Boolean, default - false. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.

    backlogLimit

    Integer, default - none. Configures the backlog to hold records. If during boot audit=1 is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.

    maxLogFile

    Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.

    maxLogFileAction

    String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:

    • keep_logs - rotate logs but never delete them

    • rotate - add a cron job to compress rotated log files and keep maximum 5 compressed files.

    • compress - compress log files and keep them under the /var/log/auditd/ directory. Requires auditd_max_log_file_keep to be enabled.

    CIS rule: 4.1.2.2.

    maxLogFileKeep

    Integer, default - 5. Defines the number of compressed log files to keep under the /var/log/auditd/ directory. Requires auditd_max_log_file_action=compress. CIS rules - none.

    mayHaltSystem

    Boolean, default - false. Halts the system when the audit logs are full. Applies the following configuration:

    • space_left_action = email

    • action_mail_acct = root

    • admin_space_left_action = halt

    CIS rule: 4.1.2.3.

    customRules

    String, default - none. Base64-encoded content of the 60-custom.rules file for any architecture. CIS rules - none.

    customRulesX32

    String, default - none. Base64-encoded content of the 60-custom.rules file for the i386 architecture. CIS rules - none.

    customRulesX64

    String, default - none. Base64-encoded content of the 60-custom.rules file for the x86_64 architecture. CIS rules - none.

    presetRules

    String, default - none. Comma-separated list of the following built-in preset rules:

    • access

    • actions

    • delete

    • docker

    • identity

    • immutable

    • logins

    • mac-policy

    • modules

    • mounts

    • perm-mod

    • privileged

    • scope

    • session

    • system-locale

    • time-change

    You can use two keywords for these rules:

    • none - disables all built-in rules.

    • all - enables all built-in rules. With this key, you can add the ! prefix to a rule name to exclude some rules. You can use the ! prefix for rules only if you add the all keyword as the first rule. Place a rule with the ! prefix only after the all keyword.

    Example configurations:

    • presetRules: none - disable all preset rules

    • presetRules: docker - enable only the docker rules

    • presetRules: access,actions,logins - enable only the access, actions, and logins rules

    • presetRules: all - enable all preset rules

    • presetRules: all,!immutable,!sessions - enable all preset rules except immutable and sessions


    CIS controls
    4.1.3 (time-change)
    4.1.4 (identity)
    4.1.5 (system-locale)
    4.1.6 (mac-policy)
    4.1.7 (logins)
    4.1.8 (session)
    4.1.9 (perm-mod)
    4.1.10 (access)
    4.1.11 (privileged)
    4.1.12 (mounts)
    4.1.13 (delete)
    4.1.14 (scope)
    4.1.15 (actions)
    4.1.16 (modules)
    4.1.17 (immutable)
    Docker CIS controls
    1.1.4
    1.1.8
    1.1.10
    1.1.12
    1.1.13
    1.1.15
    1.1.16
    1.1.17
    1.1.18
    1.2.3
    1.2.4
    1.2.5
    1.2.6
    1.2.7
    1.2.10
    1.2.11

    See also

    Operations Guide: Troubleshooting - The auditd events cause ‘backlog limit exceeded’ messages

  13. Optional. Configure external identity provider for IAM.

  14. Optional. Enable infinite timeout for all bootstrap stages by exporting the following environment variable or adding it to bootstrap.env:

    export KAAS_BOOTSTRAP_INFINITE_TIMEOUT=true

    Infinite timeout prevents the bootstrap failure due to timeout. This option is useful in the following cases:

    • The network speed is slow for artifacts downloading

    • An infrastructure configuration does not allow booting fast

    • A bare-metal node inspecting presupposes more than two HDDSATA disks to attach to a machine

  15. Optional. Available since Container Cloud 2.23.0. Customize the cluster and region name by exporting the following environment variables or adding them to bootstrap.env:

    export REGION=<customRegionName>
export CLUSTER_NAME=<customClusterName>

    By default, the system uses region-one for the region name and kaas-mgmt for the management cluster name.

  16. Run the bootstrap script:

    ./bootstrap.sh all

    • In case of deployment issues, refer to Troubleshooting and inspect logs.

    • If the script fails for an unknown reason:

      1. Run the cleanup script:

        ./bootstrap.sh cleanup

      2. Rerun the bootstrap script.

  17. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private ssh_key for access to the management cluster nodes that is located in the same directory as the bootstrap script.

      Note

      If the initial version of your Container Cloud management cluster was earlier than 2.6.0, ssh_key is named openstack_tmp and is located at ~/.ssh/.

    • The URL for the Container Cloud web UI.

      To create users with permissions required for accessing the Container Cloud web UI, see Create initial users after a management cluster bootstrap.

    • The StackLight endpoints. For details, see Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) and communicate with Keycloak to authenticate users. Keycloak is exposed using HTTPS and self-signed TLS certificates that are not trusted by web browsers.

    To use your own TLS certificates for Keycloak, refer to Configure TLS certificates for cluster applications.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.

  18. Verify that network addresses used on your clusters do not overlap with the following default MKE network addresses for Swarm and MCR:

    • 10.0.0.0/16 is used for Swarm networks. IP addresses from this network are virtual.

    • 10.99.0.0/16 is used for MCR networks. IP addresses from this network are allocated on hosts.

    Verification of Swarm and MCR network addresses

    To verify Swarm and MCR network addresses, run on any master node:

    docker info

    Example of system response:

    Server:
 ...
 Swarm:
  ...
  Default Address Pool: 10.0.0.0/16
  SubnetSize: 24
  ...
 Default Address Pools:
   Base: 10.99.0.0/16, Size: 20
 ...

    Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress network is created by default and occupies the 10.0.0.0/24 address block. Also, three MCR networks are created by default and occupy three address blocks: 10.99.0.0/20, 10.99.16.0/20, 10.99.32.0/20.

    To verify the actual networks state and addresses in use, run:

    docker network ls
docker network inspect <networkName>

  19. Optional. Deploy an additional regional cluster as described in Deploy an additional regional cluster (optional).

Now, you can proceed with operating your management cluster using the Container Cloud web UI and deploying managed clusters as described in Create and operate an OpenStack-based managed cluster.

See also