Bootstrap a management cluster

Bootstrap a management cluster

This section describes how to prepare and bootstrap a baremetal-based management cluster. The procedure includes:

  • A runbook that describes how to create a seed node that is a temporary server used to run the management cluster bootstrap scripts.

  • A step-by-step instruction how to prepare metadata for the bootstrap scripts and how to run them.

Prepare the seed node

Before installing Mirantis Container Cloud on a bare metal environment, complete the following preparation steps:

  1. Verify that the hardware allocated for the installation meets the minimal requirements described in Reference Architecture: Requirements for a baremetal-based Container Cloud.

  2. Install basic Ubuntu 18.04 server using standard installation images of the operating system on the bare metal seed node.

  3. Log in to the seed node that is running Ubuntu 18.04.

  4. Create a virtual bridge to connect to your PXE network on the seed node. Use the following netplan-based configuration file as an example:

    # cat /etc/netplan/config.yaml
    network:
      version: 2
      renderer: networkd
      ethernets:
        ens3:
            dhcp4: false
            dhcp6: false
      bridges:
          br0:
              addresses:
              # Please, adjust for your environment
              - 10.0.0.15/24
              dhcp4: false
              dhcp6: false
              # Please, adjust for your environment
              gateway4: 10.0.0.1
              interfaces:
              # Interface name may be different in your environment
              - ens3
              nameservers:
                  addresses:
                  # Please, adjust for your environment
                  - 8.8.8.8
              parameters:
                  forward-delay: 4
                  stp: false
    
  5. Apply the new network configuration using netplan:

    sudo netplan apply
    
  6. Verify the new network configuration:

    sudo brctl show
    

    Example of system response:

    bridge name     bridge id               STP enabled     interfaces
    br0             8000.fa163e72f146       no              ens3
    

    Verify that the interface connected to the PXE network belongs to the previously configured bridge.

  7. Install the current Docker version available for Ubuntu 18.04:

    sudo apt install docker.io
    
  8. Verify that your logged USER has access to the Docker daemon:

    sudo usermod -aG docker $USER
    
  9. Log out and log in again to the seed node to apply the changes.

  10. Verify that Docker is configured correctly and has access to Container Cloud CDN. For example:

    docker run --rm alpine sh -c "apk add --no-cache curl; \
    curl https://binary.mirantis.com"
    

    The system output must contain a json file with no error messages. In case of errors, follow the steps provided in Troubleshooting.

Proceed with Verify the seed node.

Verify the seed node

Before you proceed to bootstrapping the management cluster on bare metal, perform the following steps:

  1. Verify that the seed node has direct access to the Baseboard Management Controller (BMC) of each baremetal host. All target hardware nodes must be in the power off state.

    For example, using the IPMI tool:

    ipmitool -I lanplus -H 'IPMI IP' -U 'IPMI Login' -P 'IPMI password' \
    chassis power status
    

    Example of system response:

    Chassis Power is off
    
  2. Verify that you configured each bare metal host as follows:

    • Enable the boot NIC support for UEFI load. Usually, at least the built-in network interfaces support it.

    • Enable the UEFI-LAN-OPROM support in BIOS -> Advanced -> PCIPCIe.

    • Enable the IPv4-PXE stack.

    • Set the following boot order:

      1. UEFI-DISK

      2. UEFI-PXE

    • If your PXE network is not configured to use the first network interface, fix the UEFI-PXE boot order to speed up node discovering by selecting only one required network interface.

    • Power off all bare metal hosts.

    Warning

    Only one Ethernet port on a host must be connected to the Common/PXE network at any given time. The physical address (MAC) of this interface must be noted and used to configure the BareMetalHost object describing the host.

Proceed with Prepare metadata and deploy the management cluster.

Prepare metadata and deploy the management cluster

Using the example procedure below, replace the addresses and credentials in the configuration YAML files with the data from your environment. Keep everything else as is, including the file names and YAML structure.

The overall network mapping scheme with all L2 parameters, for example, for a single 10.0.0.0/24 network, is described in the following table. The configuration of each parameter indicated in this table is described in the steps below.

Network mapping overview

Deployment file name

Parameters and values

cluster.yaml

  • SET_LB_HOST=10.0.0.90

  • SET_METALLB_ADDR_POOL=10.0.0.61-10.0.0.80

ipam-objects.yaml

  • SET_IPAM_CIDR=10.0.0.0/24

  • SET_PXE_NW_GW=10.0.0.1

  • SET_PXE_NW_DNS=8.8.8.8

  • SET_IPAM_POOL_RANGE=10.0.0.100-10.0.0.252

  • SET_LB_HOST=10.0.0.90

  • SET_METALLB_ADDR_POOL=10.0.0.61-10.0.0.80

bootstrap.sh

  • KAAS_BM_PXE_IP=10.0.0.20

  • KAAS_BM_PXE_MASK=24

  • KAAS_BM_PXE_BRIDGE=br0

  • KAAS_BM_BM_DHCP_RANGE=10.0.0.30,10.0.0.49

  • KEYCLOAK_FLOATING_IP=10.0.0.70

  • IAM_FLOATING_IP=10.0.0.71

  • PROXY_FLOATING_IP=10.0.0.72


  1. Log in to the seed node that you configured as described in Prepare the seed node.

  2. Change to your preferred work directory, for example, your home directory:

    cd $HOME
    
  3. Download and run the Container Cloud bootstrap script to this directory:

    wget https://binary.mirantis.com/releases/get_container_cloud.sh
    chmod 0755 get_container_cloud.sh
    ./get_container_cloud.sh
    
  4. Change the directory to the kaas-bootstrap folder created by the get_container_cloud.sh script:

    cd kaas-bootstrap
    
  5. Obtain your license file that will be required during the bootstrap. See step 3 in Getting Started with Mirantis Container Cloud.

  6. Save the license file as mirantis.lic under the kaas-bootstrap directory.

  7. Create a copy of the current templates directory for future reference.

    mkdir templates.backup
    cp -r templates/*  templates.backup/
    
  8. Update the cluster definition template in templates/bm/cluster.yaml.template according to the environment configuration. Use the table below. Manually set all parameters that start with SET_. For example, SET_METALLB_ADDR_POOL.

    Cluster template mandatory parameters

    Parameter

    Description

    Example value

    SET_LB_HOST

    The IP address of the externally accessible API endpoint of the management cluster. This address must NOT be within the SET_METALLB_ADDR_POOL range but must be from the PXE network. External load balancers are not supported.

    10.0.0.90

    SET_METALLB_ADDR_POOL

    The IP range to be used as external load balancers for the Kubernetes services with the LoadBalancer type. This range must be within the PXE network. The minimum required range is 19 IP addresses.

    10.0.0.61-10.0.0.80

  9. Optional. Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

    In templates/bm/cluster.yaml.template, add the ntp:servers section with the list of required servers names:

    spec:
      ...
      providerSpec:
        value:
          kaas:
          ...
            regional:
              - helmReleases:
                - name: baremetal-provider
                  values:
                    config:
                      lcm:
                        ...
                        ntp:
                          servers:
                          - 0.pool.ntp.org
                          ...
                provider: baremetal
                ...
    
  10. Inspect the default bare metal host profile definition in templates/bm/baremetalhostprofiles.yaml.template. If your hardware configuration differs from the reference, adjust the default profile to match. For details, see Customize the default bare metal host profile.

  11. Update the bare metal hosts definition template in templates/bm/baremetalhosts.yaml.template according to the environment configuration. Use the table below. Manually set all parameters that start with SET_.

    Bare metal hosts template mandatory parameters

    Parameter

    Description

    Example value

    SET_MACHINE_0_IPMI_USERNAME

    The IPMI user name in the base64 encoding to access the BMC. 0

    dXNlcg== (base64 encoded user)

    SET_MACHINE_0_IPMI_PASSWORD

    The IPMI password in the base64 encoding to access the BMC. 0

    cGFzc3dvcmQ= (base64 encoded password)

    SET_MACHINE_0_MAC

    The MAC address of the first management master node in the PXE network.

    ac:1f:6b:02:84:71

    SET_MACHINE_0_BMC_ADDRESS

    The IP address of the BMC endpoint for the first master node in the management cluster. Must be an address from the OOB network that is accessible through the PXE network default gateway.

    192.168.100.11

    SET_MACHINE_1_IPMI_USERNAME

    The IPMI user name in the base64 encoding to access the BMC. 0

    dXNlcg== (base64 encoded user)

    SET_MACHINE_1_IPMI_PASSWORD

    The IPMI password in the base64 encoding to access the BMC. 0

    cGFzc3dvcmQ= (base64 encoded password)

    SET_MACHINE_1_MAC

    The MAC address of the second management master node in the PXE network.

    ac:1f:6b:02:84:72

    SET_MACHINE_1_BMC_ADDRESS

    The IP address of the BMC endpoint for the second master node in the management cluster. Must be an address from the OOB network that is accessible through the PXE network default gateway.

    192.168.100.12

    SET_MACHINE_2_IPMI_USERNAME

    The IPMI user name in the base64 encoding to access the BMC. 0

    dXNlcg== (base64 encoded user)

    SET_MACHINE_2_IPMI_PASSWORD

    The IPMI password in the base64 encoding to access the BMC. 0

    cGFzc3dvcmQ= (base64 encoded password)

    SET_MACHINE_2_MAC

    The MAC address of the third management master node in the PXE network.

    ac:1f:6b:02:84:73

    SET_MACHINE_2_BMC_ADDRESS

    The IP address of the BMC endpoint for the third master node in the management cluster. Must be an address from the OOB network that is accessible through the PXE network default gateway.

    192.168.100.13

    0(1,2,3,4,5,6)

    You can obtain the base64-encoded user name and password using the following command in your Linux console:

    $ echo -n <username|password> | base64
    
  12. Update the IP address pools definition template in templates/bm/ipam-objects.yaml.template according to the environment configuration. Use the table below. Manually set all parameters that start with SET_. For example, SET_IPAM_POOL_RANGE.

    IP address pools template mandatory parameters

    Parameter

    Description

    Example value

    SET_IPAM_CIDR

    The address of PXE network in CIDR notation. Must be minimum in the /24 network.

    10.0.0.0/24

    SET_PXE_NW_GW

    The default gateway in the PXE network. Since this is the only network that Container Cloud will use, this gateway must provide access to:

    • The Internet to download the Mirantis artifacts

    • The OOB network of the Container Cloud cluster

    10.0.0.1

    SET_PXE_NW_DNS

    An external (non-Kubernetes) DNS server accessible from the PXE network. This server will be used by the bare metal hosts in all Container Cloud clusters.

    8.8.8.8

    SET_IPAM_POOL_RANGE

    This pool range includes addresses that will be allocated to bare metal hosts in all Container Cloud clusters. The size of this range limits the number of hosts that can be deployed by the instance of Container Cloud.

    10.0.0.100-10.0.0.252

    SET_LB_HOST 1

    The IP address of the externally accessible API endpoint of the management cluster. This address must NOT be within the SET_METALLB_ADDR_POOL range but must be from the PXE network. External load balancers are not supported.

    10.0.0.90

    SET_METALLB_ADDR_POOL 1

    The IP range to be used as external load balancers for the Kubernetes services with the LoadBalancer type. This range must be within the PXE network. The minimum required range is 19 IP addresses.

    10.0.0.61-10.0.0.80

    1(1,2)

    Use the same value that you used for this parameter in the cluster.yaml.template file (see above).

  13. Available since 2.6.0 Optional. If you require all Internet access to go through a proxy server, in bootstrap.env, add the following environment variables to bootstrap the management and regional cluster using proxy:

    • HTTP_PROXY

    • HTTPS_PROXY

    • NO_PROXY

    Example snippet:

    export HTTP_PROXY=http://proxy.example.com:3128
    export HTTPS_PROXY=http://user:pass@proxy.example.com:3128
    export NO_PROXY=172.18.10.0,registry.internal.lan
    

    The following variables formats are accepted:

    Proxy configuration data

    Variable

    Format

    • HTTP_PROXY

    • HTTPS_PROXY

    • http://proxy.example.com:port - for anonymous access

    • http://user:password@proxy.example.com:port - for restricted access

    • NO_PROXY

    Comma-separated list of IP addresses or domain names

    For the list of Mirantis resources and IP addresses to be accessible from the Container Cloud clusters, see Reference Architecture: Hardware and system requirements.

  14. Optional. Skip this step to use the default password password in the Container Cloud web UI.

    Caution

    For security reasons, Mirantis strongly recommends changing the default password on publicly accessible Container Cloud deployments.

    Configure the IAM parameters:

    1. Create hashed passwords for every IAM role: reader, writer, and operator for bare metal deployments:

      ./bin/hash-generate -i 27500
      

      The hash-generate utility requests you to enter a password and outputs the parameters required for the next step. Save the password that you enter in a secure location. This password will be used to access the Container Cloud web UI with a specific IAM role.

      Example of system response:

      passwordSalt: 6ibPZdUfQK8PsOpSmyVJnA==
      passwordHash: 23W1l65FBdI3NL7LMiUQG9Cu62bWLTqIsOgdW8xNsqw=
      passwordHashAlgorithm: pbkdf2-sha256
      passwordHashIterations: 27500
      

      Run the tool several times to generate hashed passwords for every IAM role.

    2. Open templates/cluster.yaml.template for editing.

    3. In the initUsers section, add the following parameters for each IAM role that you generated in the previous step:

      • passwordSalt - base64-encoded randomly generated sequence of bytes.

      • passwordHash - base64-encoded password hash generated using passwordHashAlgorithm with passwordHashIterations. Supported algorithms include pbkdf2-sha256 and pbkdf-sha512.

  15. Optional. Configure external identity provider for IAM.

  16. Configure the Ceph cluster:

    1. Optional. Available since 2.6.0, Technology Preview. Configure Ceph controller to manage Ceph nodes resources. In templates/bm/cluster.yaml.template, in the ceph-controller section of spec.providerSpec.value.helmReleases, specify the hyperconverge parameter with required resource requests, limits, or tolerations:

      spec:
         providerSpec:
           value:
             helmReleases:
             - name: ceph-controller
               values:
                 hyperconverge:
                   tolerations: <ceph tolerations map>
                   resources: <ceph resource management map>
      

      For the parameters description, see Operations Guide: Enable Ceph tolerations and resources management.

    2. In templates/bm/kaascephcluster.yaml.template:

      • Configure dedicated networks clusterNet and publicNet for Ceph components.

      • Set up the disk configuration according to your hardware node specification. Verify that the storageDevices section has a valid list of HDD device names and each device is empty, that is, no file system is present on it. To enable all LCM features of Ceph controller, set manageOsds to true.

      • If required, configure other parameters as described in Operations Guide: Ceph advanced configuration.

      Configuration example:

      manageOsds: true
      ...
      # This part of KaaSCephCluster should contain valid networks definition
      network:
        clusterNet: 10.10.10.0/24
        publicNet: 10.10.11.0/24
      ...
      nodes:
        master-0:
        ...
        <node_name>:
          ...
          # This part of KaaSCephCluster should contain valid device names
          storageDevices:
          - name: sdb
            config:
              deviceClass: hdd
          # Each storageDevices dicts can have several devices
          - name: sdc
            config:
              deviceClass: hdd
          # All devices for Ceph also should be described to ``wipe`` in
          # ``baremetalhosts.yaml.template``
          - name: sdd
            config:
             deviceClass: hdd
          # Do not to include first devices here (like vda or sda)
          # because they will be allocated for operating system
      
    3. In machines.yaml.template, verify that the metadata:name structure matches the machine names in the spec:nodes structure of kaascephcluster.yaml.template.

  17. Verify that the kaas-bootstrap directory contains the following files:

    # tree  ~/kaas-bootstrap
      ~/kaas-bootstrap/
      ....
      ├── bootstrap.sh
      ├── kaas
      ├── mirantis.lic
      ├── releases
      ...
      ├── templates
      ....
      │   ├── bm
      │   │   ├── baremetalhostprofiles.yaml.template
      │   │   ├── baremetalhosts.yaml.template
      │   │   ├── cluster.yaml.template
      │   │   ├── ipam-objects.yaml.template
      │   │   ├── kaascephcluster.yaml.template
      │   │   └── machines.yaml.template
      ....
      ├── templates.backup
          ....
    
  18. Export all required parameters using the table below.

    export KAAS_BM_ENABLED="true"
    #
    export KAAS_BM_PXE_IP="10.0.0.20"
    export KAAS_BM_PXE_MASK="24"
    export KAAS_BM_PXE_BRIDGE="br0"
    #
    export KAAS_BM_BM_DHCP_RANGE="10.0.0.30,10.0.0.49"
    #
    export KEYCLOAK_FLOATING_IP="10.0.0.70"
    export IAM_FLOATING_IP="10.0.0.71"
    export PROXY_FLOATING_IP="10.0.0.72"
    unset KAAS_BM_FULL_PREFLIGHT
    
    Bare metal prerequisites data

    Parameter

    Description

    Example value

    KAAS_BM_PXE_IP

    The provisioning IP address. This address will be assigned to the interface of the seed node defined by the KAAS_BM_PXE_BRIDGE parameter (see below). The PXE service of the bootstrap cluster will use this address to network boot the bare metal hosts for the management cluster.

    10.0.0.20

    KAAS_BM_PXE_MASK

    The CIDR prefix for the PXE network. It will be used with all of the addresses below when assigning them to interfaces.

    24

    KAAS_BM_PXE_BRIDGE

    The PXE network bridge name. The name must match the name of the bridge created on the seed node during the Prepare the seed node stage.

    br0

    KAAS_BM_BM_DHCP_RANGE

    The start_ip and end_ip addresses must be within the PXE network. This range will be used by dnsmasq to provide IP addresses for nodes during provisioning.

    10.0.0.30,10.0.0.49

    KEYCLOAK_FLOATING_IP

    The spec.loadBalancerIP address for the Keycloak service. Use the address from the top of the SET_METALLB_ADDR_POOL range.

    10.0.0.70 2

    IAM_FLOATING_IP

    The spec.loadBalancerIP address for the IAM service. Use the address from the top of the SET_METALLB_ADDR_POOL range.

    10.0.0.71 2

    PROXY_FLOATING_IP

    The spec.loadBalancerIP address for the Squid service. Use the address from the top of the SET_METALLB_ADDR_POOL range.

    10.0.0.72 2

    KAAS_BM_FULL_PREFLIGHT

    The verification preflight check to validate the deployment before bootstrap:

    • If set to true, the full preflight command will run up to 15 minutes to simulate the main stages of the management cluster deployment and to verify that the cluster network and nodes in the BareMetalHost template are configured correctly.

    • If unset using unset KAAS_BM_FULL_PREFLIGHT, the fast preflight command will execute a quick IPMI check to ensure that the template parameters for the Baseboard Management Controller (BMC) access credentials and PXE service are configured correctly.

    unset KAAS_BM_FULL_PREFLIGHT

    2(1,2,3)

    Must not conflict with other *_FLOATING_IP parameters. Use the address from the top of the SET_METALLB_ADDR_POOL range.

  19. Run the verification preflight script to validate the deployment templates configuration:

    ./bootstrap.sh preflight
    

    The command outputs a human-readable report with the verification details:

    • If you run the full preflight command, the report includes information whether the nodes successfully passed the inspection stage and outputs the ICMP results of the networks verification.

      The following example illustrates a positive system response where each node IP and seed addresses have dict with the standard ICMP result:

      {"10.0.0.20":{"packet_loss":"0","packets_received":"10","packets_transmitted":"10","roundtrip_avg":"2.2381ms","roundtrip_max"      :"12.144668ms","roundtrip_min":"884.641Вµs","roundtrip_stddev":"3.31024ms"},\
      "10.0.0.200":{"packet_loss":"0","packets_received      ":"10","packets_transmitted":"10","roundtrip_avg":"4.066847ms","roundtrip_max":"12.185964ms","roundtrip_min":"1.146483ms","ro      undtrip_stddev":"4.267026ms"},\
      "10.0.0.201":{"packet_loss":"0","packets_received":"10","packets_transmitted":"10","roundtrip_a      vg":"3.658907ms","roundtrip_max":"19.389166ms","roundtrip_min":"1.286625ms","roundtrip_stddev":"5.285539ms"},\
      "10.0.0.202":{"p      acket_loss":"0","packets_received":"10","packets_transmitted":"10","roundtrip_avg":"411.068Вµs","roundtrip_max":"577.381Вµs",      "roundtrip_min":"302.042Вµs","roundtrip_stddev":"92.759Вµs"}}
      

      The following example illustrates a negative system response when a node is unreachable:

      min package test: ping loss 100.0 for 10.0.0.20 higher than acceptable package loss 30.0
      
    • If you run the fast preflight command, the report includes the list of verified bare metal nodes and their Chassis Power status. This status is based on the deployment templates configuration used during the verification.

    Caution

    If the report contains information about missing dependencies or incorrect configuration, fix the issues before proceeding to the next step.

  20. Run the bootstrap script:

    ./bootstrap.sh all
    

    Warning

    During the bootstrap process, do not manually restart or power off any of the bare metal hosts.

  21. When the bootstrap is complete, collect and save the following management cluster details in a secure location:

    • The kubeconfig file located in the same directory as the bootstrap script. This file contains the admin credentials for the management cluster.

    • The private ssh_key for access to the management cluster nodes that is located in the same directory as the bootstrap script.

    • The URL and credentials for the Container Cloud web UI. The system outputs these details when the bootstrap completes.

    • The StackLight endpoints. For details, see Operations Guide: Access StackLight web UIs.

    • The Keycloak URL that the system outputs when the bootstrap completes. The admin password for Keycloak is located in kaas-bootstrap/passwords.yml along with other IAM passwords.

    Note

    When the bootstrap is complete, the bootstrap cluster resources are freed up.