MCP Deployment Guide¶

Create a project repository¶

An MCP cluster deployment configuration is stored in a Git repository created on a per-customer basis. This section instructs you on how to manually create and prepare your project repository for an MCP deployment.

Before you start this procedure, create a Git repository in your version control system, such as GitHub.

To create a project repository manually:

1. Log in to any computer.

2. Create an empty directory and change to that directory. In the example below, it is mcpdoc.

3. Initialize your project repository:

   git init
   

   Example of system response:

   Initialized empty Git repository in /Users/crh/Dev/mcpdoc/.git/
   

4. Add your repository to the directory you have created:

   git remote add origin <YOUR-GIT-REPO-URL>
   

5. Verify that Git and your local repository are set up correctly by creating and pushing a test file to your project repository. Run the following example commands:

   Note

   The example commands below require the Git and GitHub credentials to be created and configured for your project repository.

   git remote add origin https://github.com/example_account/mcpdoc.git
   git config --local user.email "example@example.com"
   git config --local user.name "example_gituser"
   git config -l
   
   echo "#. mcpdoc" >> README.md
   git add README.md
   git commit -m "first commit"
   git push -u origin master
   

6. Create the following directories for your deployment metadata model:

   mkdir -p classes/cluster
   mkdir nodes
   

7. Add the Reclass variable to your bash profile by verifying your current directory using pwd and adding the string that exports the Reclass variable with the output value of the pwd command:

   pwd
   vim ~/.bash_profile
   export RECLASS_REPO=<PATH_TO_YOUR_DEV_DIRECTORY>
   

   Example of system response:

   /Users/crh/Dev/mcpdoc/
   
   "~/.bash_profile" 13L, 450C
   export RECLASS_REPO="/Users/crh/Dev/mcpdoc/"
   

8. Log out and log back in.

9. Verify that your ~/.bash_profile is sourced:

   echo $RECLASS_REPO
   

   The command must show the value of your RECLASS_REPO variable.

   Example of system response:

   /Users/crh/Dev/mcpdoc/
   

10. Add the Mirantis Reclass module to your repository as a submodule:

    git submodule add https://github.com/Mirantis/reclass-system-salt-model ./classes/system/
    

    Example of system response:

    Cloning into '<PATH_TO_YOUR_DEV_DIRECTORY>/classes/system'...
    remote: Counting objects: 8923, done.
    remote: Compressing objects: 100% (214/214), done.
    remote: Total 8923 (delta 126), reused 229 (delta 82), pack-reused 8613
    Receiving objects: 100% (8923/8923), 1.15 MiB | 826.00 KiB/s, done.
    Resolving deltas: 100% (4482/4482), done.
    Checking connectivity... done.
    

11. Update the submodule:

    git submodule sync
    git submodule update --init --recursive --remote
    

12. Add your changes to a new commit:

    git add -A
    

13. Commit your changes:

    git commit
    

14. Add your commit message.

    Example of system response:

    [master (root-commit) 9466ada] Initial Commit
     2 files changed, 4 insertions(+)
     create mode 100644 .gitmodules
     create mode 160000 classes/system
    

15. Push your changes:

    git push
    

16. Proceed to Create a deployment metadata model.

Create a deployment metadata model¶

In a Reclass metadata infrastructural model, the data is stored as a set of several layers of objects, where objects of a higher layer are combined with objects of a lower layer, that allows for as flexible configuration as required.

The MCP metadata model has the following levels:

• Service level includes metadata fragments for individual services that are stored in Salt formulas and can be reused in multiple contexts.
• System level includes sets of services combined in a such way that the installation of these services results in a ready-to-use system.
• Cluster level is a set of models that combine already created system objects into different solutions. The cluster module settings override any settings of service and system levels and are specific for each deployment.

The model layers are firmly isolated from each other. They can be aggregated on a south-north direction using service interface agreements for objects on the same level. Such approach allows reusing of the already created objects both on service and system levels.

This section describes how to generate the cluster level metadata model for your MCP cluster deployment using the Model Designer web UI. The tool used to generate the model is Cookiecutter, a command-line utility that creates projects from templates.

While generating a metadata model, you can enable automated encryption of all secrets for the Salt Master node .iso file.

The workflow of a model creation includes the following stages:

1. Defining the model through the Model Designer web UI.

2. Optional. Tracking the execution of the model creation pipeline in the Jenkins web UI.

3. Obtaining the generated model to your email address or getting it published to the project repository directly.

   Note

   If you prefer publishing to the project repository, verify that the dedicated repository is configured correctly and Jenkins can access it. See Create a project repository for details.

As a result, you get a generated deployment model and can customize it to fit specific use-cases. Otherwise, you can proceed with the base infrastructure deployment.

git add *
git commit –m "Initial commit"

git pull -r
git push --set-upstream origin master

Prerequisites for MCP DriveTrain deployment¶

Before you proceed with the actual deployment, verify that you have performed the following steps:

1. Deploy the Foundation physical node using one of the initial versions of Ubuntu Xenial, for example, 16.04.1.

   Use any standalone hardware node where you can run a KVM-based day01 virtual machine with an access to the deploy/control network. The Foundation node will host the Salt Master node that also includes the MAAS provisioner by default. For the offline case deployment, the Foundation node will also host the mirror VM.

2. Depending on your case, proceed with one of the following options:

   • If you do not have a deployment metadata model:

     1. Create a model using the Model Designer UI as described in Create a deployment metadata model.

        Note

        For an offline deployment, select the Offline deployment and Local repositories options under the Repositories section on the Infrastructure parameters tab.

     2. Customize the obtained configuration drives as described in Generate configuration drives manually. For example, enable custom user access.

   • If you use an already existing model that does not have configuration drives, or you want to generate updated configuration drives, proceed with Generate configuration drives manually.

3. Configure the following bridges on the Foundation node: br-mgm for the management network and br-ctl for the control network.

   1. Log in to the Foundation node through IPMI.

      Note

      If the IPMI network is not reachable from the management or control network, add the br-ipmi bridge for the IPMI network or any other network that is routed to the IPMI network.

   2. Create PXE bridges to provision network on the foundation node:

      brctl addbr br-mgm
      brctl addbr br-ctl
      

   3. Install the br-ctl utility:

      apt install bridge-utils
      

   4. Add the bridges definition for br-mgm and br-ctl to /etc/network/interfaces. Use definitions from your deployment metadata model.

      Example:

      auto br-mgm
      iface br-mgm inet static
              address 172.17.17.200
              netmask 255.255.255.192
              bridge_ports bond0
      

   5. Restart networking from the IPMI console to bring the bonds up.

   6. Verify that the foundation node bridges are up by checking the output of the ip a show command:

      ip a show br-ctl
      

      Example of system response:

      8: br-ctl: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
          link/ether 00:1b:21:93:c7:c8 brd ff:ff:ff:ff:ff:ff
          inet 172.17.45.241/24 brd 172.17.45.255 scope global br-ctl
             valid_lft forever preferred_lft forever
          inet6 fe80::21b:21ff:fe93:c7c8/64 scope link
             valid_lft forever preferred_lft forever
      

4. Depending on your case, proceed with one of the following options:

   • If you perform an online deployment, proceed to Deploy the Salt Master node.
   • If you perform the offline deployment or online deployment with local mirrors, proceed to Deploy the APT node.

Deploy the APT node¶

MCP enables you to deploy the whole MCP cluster without access to the Internet. On creating the metadata model, along with the configuration drive for the cfg01 VM, you will obtain a preconfigured QCOW2 image that will contain packages, Docker images, operating system images, Git repositories, and other software required specifically for the offline deployment.

This section describes how to deploy the apt01 VM using the prebuilt configuration drive.

To deploy the APT node:

1. Verify that you completed steps described in Prerequisites for MCP DriveTrain deployment.

2. Log in to the Foundation node.

   Note

   Root privileges are required for following steps. Execute the commands as a root user or use sudo.

3. Download the latest version of the prebuilt http://images.mirantis.com/mcp-offline-image-<BUILD-ID>.qcow2 image for the apt node from http://images.mirantis.com.

4. In the /var/lib/libvirt/images/ directory, create an apt01/ subdirectory where the offline mirror image will be stored:

   Note

   You can create and use a different subdirectory in /var/lib/libvirt/images/. If that is the case, verify that you specify the correct directory for the VM_*DISK variables described in next steps.

   mkdir -p /var/lib/libvirt/images/apt01/
   

5. Save the image on the Foundation node as /var/lib/libvirt/images/apt01/system.qcow2.

6. Copy the configuration ISO drive for the APT VM provided with the metadata model for the offline image to, for example, /var/lib/libvirt/images/apt01/.

   Caution

   By default, the prebuilt image does not have a possibility to log in to.

   Note

   If you are using an already existing model that does not have configuration drives, or you want to generate updated configuration drives, for example, with an unlocked root login for debugging purposes, proceed with Generate configuration drives manually.

   cp /path/to/prepared-drive/apt01-config.iso /var/lib/libvirt/images/apt01/apt01-config.iso
   

7. Deploy the APT node:

   1. Download the shell script from GitHub:

      export MCP_VERSION="master"
      wget https://raw.githubusercontent.com/Mirantis/mcp-common-scripts/${MCP_VERSION}/predefine-vm/define-vm.sh
      

   2. Make the script executable, export the required variables:

      chmod +x define-vm.sh
      export VM_NAME="apt01.<CLUSTER_DOMAIN>"
      export VM_SOURCE_DISK="/var/lib/libvirt/images/apt01/system.qcow2"
      export VM_CONFIG_DISK="/var/lib/libvirt/images/apt01/apt01-config.iso"
      

      The CLUSTER_DOMAIN value is the cluster domain name used for the model. See Basic deployment parameters for details.

      Note

      You may add other optional variables that have default values and change them depending on your deployment configuration. These variables include:

      • VM_MGM_BRIDGE_NAME="br-mgm"
      • VM_CTL_BRIDGE_NAME="br-ctl"
      • VM_MEM_KB="12589056"
      • VM_CPUS="4"

      
      

      The recommended VM_MEM_KB for the Salt Master node is 12589056 (or more depending on your cluster size) that is 12 GB of RAM. For large clusters, you should also increase VM_CPUS.

      The recommended VM_MEM_KB for the local mirror node is 8388608 (or more) that is 8 GB of RAM.

      The br-mgm and br-ctl values are the names of the Linux bridges. See Prerequisites for MCP DriveTrain deployment for details. Custom names can be passed to a VM definition using the VM_MGM_BRIDGE_NAME and VM_CTL_BRIDGE_NAME variables accordingly.

   3. Run the shell script:

      ./define-vm.sh
      

8. Start the apt01 VM:

   virsh start apt01.<CLUSTER_DOMAIN>
   

9. For MCP versions prior to the 2019.2.14 maintenance update, perform the following additional steps:

   1. SSH to the apt01 node.

   2. Verify the certificate:

      openssl x509 -checkend 1 -in /var/lib/docker/swarm/certificates/swarm-node.crt
      

      If the certificate has expired, restart Docker Swarm to regenerate it:

      systemctl stop docker || true
      rm -rf /var/lib/docker/swarm/*
      systemctl restart docker
      sleep 5
      docker ps
      docker swarm init --advertise-addr 127.0.0.1
      sleep 5
      cd /etc/docker/compose/docker/
      docker stack deploy --compose-file docker-compose.yml docker
      sleep 5
      cd /etc/docker/compose/aptly/
      docker stack deploy --compose-file docker-compose.yml aptly
      sleep 5
      docker ps
      

After completing the steps above, you obtain the apt01 node that contains only the pre-built content. Now, you can proceed with Deploy the Salt Master node. Once you deploy the Salt Master node, you will be able to customize the content of the local mirror, as described in Customize the prebuilt mirror node.

Deploy the Salt Master node¶

The Salt Master node acts as a central control point for the clients that are called Salt minion nodes. The minions, in their turn, connect back to the Salt Master node.

This section describes how to set up a virtual machine with Salt Master, MAAS provisioner, Jenkins server, and local Git server. The procedure is applicable to both online and offline MCP deployments.

To deploy the Salt Master node:

1. Log in to the Foundation node.

   Note

   Root privileges are required for following steps. Execute the commands as a root user or use sudo.

2. In case of an offline deployment, replace the content of the /etc/apt/sources.list file with the following lines:

   deb [arch=amd64] http://<local_mirror_url>/ubuntu xenial-security main universe restricted
   deb [arch=amd64] http://<local_mirror_url>/ubuntu xenial-updates main universe restricted
   deb [arch=amd64] http://<local_mirror_url>/ubuntu xenial main universe restricted
   

3. Create a directory for the VM system disk:

   Note

   You can create and use a different subdirectory in /var/lib/libvirt/images/. If that is the case, verify that you specify the correct directory for the VM_*DISK variables described in next steps.

   mkdir -p /var/lib/libvirt/images/cfg01/
   

4. Download the day01 image for the cfg01 node:

   wget http://images.mirantis.com/cfg01-day01-<BUILD_ID>.qcow2 -O \
   /var/lib/libvirt/images/cfg01/system.qcow2
   

   Substitute <BUILD_ID> with the required MCP Build ID, for example, 2019.2.0.

5. Copy the configuration ISO drive for the cfg01 VM provided with the metadata model for the offline image to, for example, /var/lib/libvirt/images/cfg01/cfg01-config.iso.

   Note

   If you are using an already existing model that does not have configuration drives, or you want to generate updated configuration drives, for example, with an unlocked root login for debugging purposes, proceed with Generate configuration drives manually.

   Caution

   Make sure to securely back up the configuration ISO drive image. This image contains critical information required to re-install your cfg01 node in case of storage failure, including master key for all encrypted secrets in the cluster metadata model.

   Failure to back up the configuration ISO image may result in loss of ability to manage MCP in certain hardware failure scenarios.

   cp /path/to/prepared-drive/cfg01-config.iso /var/lib/libvirt/images/cfg01/cfg01-config.iso
   

6. Create the Salt Master VM domain definition using the example script:

   1. Download the shell scripts from GitHub with the required MCP release version. For example:

      export MCP_VERSION="2019.2.0"
      git clone https://github.com/Mirantis/mcp-common-scripts -b release/${MCP_VERSION}
      

   2. Make the script executable and export the required variables:

      cd mcp-common-scripts/predefine-vm/
      export VM_NAME="cfg01.[CLUSTER_DOMAIN]"
      export VM_SOURCE_DISK="/var/lib/libvirt/images/cfg01/system.qcow2"
      export VM_CONFIG_DISK="/var/lib/libvirt/images/cfg01/cfg01-config.iso"
      

      The CLUSTER_DOMAIN value is the cluster domain name used for the model. See Basic deployment parameters for details.

      Note

      You may add other optional variables that have default values and change them depending on your deployment configuration. These variables include:

      • VM_MGM_BRIDGE_NAME="br-mgm"
      • VM_CTL_BRIDGE_NAME="br-ctl"
      • VM_MEM_KB="12589056"
      • VM_CPUS="4"

      
      

      The recommended VM_MEM_KB for the Salt Master node is 12589056 (or more depending on your cluster size) that is 12 GB of RAM. For large clusters, you should also increase VM_CPUS.

      The recommended VM_MEM_KB for the local mirror node is 8388608 (or more) that is 8 GB of RAM.

      The br-mgm and br-ctl values are the names of the Linux bridges. See Prerequisites for MCP DriveTrain deployment for details. Custom names can be passed to a VM definition using the VM_MGM_BRIDGE_NAME and VM_CTL_BRIDGE_NAME variables accordingly.

   3. Run the shell script:

      ./define-vm.sh
      

7. Start the Salt Master node VM:

   virsh start cfg01.[CLUSTER_DOMAIN]
   

8. Log in to the Salt Master virsh console with the user name and password that you created in step 4 of the Generate configuration drives manually procedure:

   virsh console cfg01.[CLUSTER_DOMAIN]
   

9. If you use local repositories, verify that mk-pipelines are present in /home/repo/mk and pipeline-library is present in /home/repo/mcp-ci after cloud-init finishes. If not, fix the connection to local repositories and run the /var/lib/cloud/instance/scripts/part-001 script.

10. Verify that the following states are successfully applied during the execution of cloud-init:

    salt-call state.sls linux.system,linux,openssh,salt
    salt-call state.sls maas.cluster,maas.region,reclass
    

    Otherwise, fix the pillar and re-apply the above states.

11. In case of using kvm01 as the Foundation node, perform the following steps on it:

    1. Depending on the deployment type, proceed with one of the options below:

       • For an online deployment, add the following deb repository to /etc/apt/sources.list.d/mcp_saltstack.list:

         deb [arch=amd64] https://mirror.mirantis.com/<MCP_VERSION>/saltstack-2017.7/xenial/ xenial main
         

       • For an offline deployment or local mirrors case, in /etc/apt/sources.list.d/mcp_saltstack.list, add the following deb repository:

         deb [arch=amd64] http://<local_mirror_url>/<MCP_VERSION>/saltstack-2017.7/xenial/ xenial main
         

    2. Install the salt-minion package.

    3. Modify /etc/salt/minion.d/minion.conf:

       id: <kvm01_FQDN>
       master: <Salt_Master_IP_or_FQDN>
       

    4. Restart the salt-minion service:

       service salt-minion restart
       

    5. Check the output of salt-key command on the Salt Master node to verify that the minion ID of kvm01 is present.

Verify the Salt infrastructure¶

Before you proceed with the deployment, validate the Reclass model and node pillars.

To verify the Salt infrastructure:

1. Log in to the Salt Master node.

2. Verify the Salt Master pillars:

   reclass -n cfg01.<cluster_domain>
   

   The cluster_domain value is the cluster domain name that you created while preparing your deployment metadata model. See Basic deployment parameters for details.

3. Verify that the Salt version for the Salt minions is the same as for the Salt Master node, that is currently 2017.7:

   salt-call --version
   salt '*' test.version
   

4. If required, enable management of the offline mirror VM (apt01) and customize the VM contents as described in Enable the management of the APT node through the Salt Master node.

Enable the management of the APT node through the Salt Master node¶

In compliance with the security best practices, MCP enables you to connect your offline mirror APT VM to the Salt Master node and manage it as any infrastructure VM on your MCP deployment.

For the deployments managed by the MCP 2018.8.0 Build ID or later, you should not manually enable the Salt minion on the offline image VM as it is configured automatically on boot during the APT VM provisioning.

Though, if your want to enable the management of the offline image VM through the Salt Master node on an existing deployment managed by the MCP version below the 2018.8.0 Build ID, you need to perform the procedure included in this section.

To enable the Salt minion on an existing offline mirror node:

1. Connect to the serial console of your offline image VM, which is included in the pre-built offline APT QCOW image:

   virsh console $(virsh list --all --name | grep ^apt01) --force
   

   Log in with the user name and password that you created in step 4 of the Generate configuration drives manually procedure.

   Example of system response:

   Connected to domain apt01.example.local
   Escape character is ^]
   

2. Press Enter to drop into the root shell.

3. Configure the Salt minion and start it:

   echo "" > /etc/salt/minion
   echo "master: <IP_address>" > /etc/salt/minion.d/minion.conf
   echo "id: <apt01.example.local>" >> /etc/salt/minion.d/minion.conf
   service salt-minion stop
   rm -f /etc/salt/pki/minion/*
   service salt-minion start
   

4. Quit the serial console by sending the Ctrl + ] combination.

5. Log in to the Salt Master node.

6. Verify that you have the offline mirror VM Salt minion connected to your Salt Master node:

   salt-key -L | grep apt
   

   The system response should include your offline mirror VM domain name. For example:

   apt01.example.local
   

7. Verify that you can access the Salt minion from the Salt Master node:

   salt apt01\* test.ping
   

8. Verify the Salt states are mapped to the offline mirror VM:

   salt apt01\* state.show_top
   

Now, you can manage your offline mirror APT VM from the Salt Master node. At this point, the Salt Master node does not manage the offline mirror content. If you need to adjust the content of your offline mirror, refer to Customize the prebuilt mirror node.

Configure MAAS for bare metal provisioning¶

Before you proceed with provisioning of the remaining bare metal nodes, configure MAAS as described below.

To configure MAAS for bare metal provisioning:

1. Log in to the MAAS web UI through http://<infra_config_deploy_address>:5240/MAAS with the following credentials:

   • Username: mirantis
   • Password: r00tme

2. Go to the Subnets tab.

3. Select the fabric that is under the deploy network.

4. In the VLANs on this fabric area, click the VLAN under the VLAN column where the deploy network subnet is.

5. In the Take action drop-down menu, select Provide DHCP.

6. Adjust the IP range as required.

   Note

   The number of IP addresses should not be less than the number of the planned VCP nodes.

7. Click Provide DHCP to submit.

8. If you use local package mirrors:

   Note

   The following steps are required only to specify the local Ubuntu package repositories that are secured by a custom GPG key and used mainly for the offline mirror images prior the MCP version 2017.12.

   1. Go to Settings > Package repositories.
   2. Click Actions > Edit on the Ubuntu archive repository.
   3. Specify the GPG key of the repository in the Key field. The key can be obtained from the aptly_gpg_public_key parameter in the cluster level Reclass model.
   4. Click Save.

Provision physical nodes using MAAS¶

Physical nodes host the Virtualized Control Plane (VCP) of your Mirantis Cloud Platform deployment.

This section describes how to provision the physical nodes using the MAAS service that you have deployed on the Foundation node while deploying the Salt Master node.

The servers that you must deploy include at least:

• For OpenStack:
  • kvm02 and kvm03 infrastructure nodes
  • cmp0 compute node
• For Kubernetes:
  • kvm02 and kvm03 infrastructure nodes
  • ctl01, ctl02, ctl03 controller nodes
  • cmp01 and cmp02 compute nodes

You can provision physical nodes automatically or manually:

• An automated provisioning requires you to define IPMI and MAC addresses in your Reclass model. After you enforce all servers, the Salt Master node commissions and provisions them automatically.
• A manual provisioning enables commissioning nodes through the MAAS web UI.

Before you proceed with the physical nodes provisioning, you may want to customize the commissioning script, for example, to set custom NIC names. For details, see: Add custom commissioning scripts.

Deploy physical nodes¶

After you provision physical nodes as described in Provision physical nodes using MAAS, follow the instruction below to deploy physical nodes intended for an OpenStack-based MCP cluster. If you plan to deploy a Kubernetes-based MCP cluster, proceed with steps 1-2 of the Kubernetes Prerequisites procedure.

linux:
  system:
    kernel:
      sysctl:
        vm.min_free_kbytes: <min_reserved_memory>

To deploy physical servers:

1. Log in to the Salt Master node.

2. Verify that the cfg01 key has been added to Salt and your host FQDN is shown properly in the Accepted Keys field in the output of the following command:

   salt-key
   

3. Verify that all pillars and Salt data are refreshed:

   salt "*" saltutil.refresh_pillar
   salt "*" saltutil.sync_all
   

4. Verify that the Reclass model is configured correctly. The following command output should show top states for all nodes:

   python -m reclass.cli --inventory
   

5. To verify that the rebooting of the nodes, which will be performed further, is successful, create the trigger file:

   salt -C 'I@salt:control or I@nova:compute or I@neutron:gateway or I@ceph:osd' \
   cmd.run "touch /run/is_rebooted"
   

6. To prepare physical nodes for VCP deployment, apply the basic Salt states for setting up network interfaces and SSH access. Nodes will be rebooted.

   Warning

   If you use kvm01 as a Foundation node, the execution of the commands below will also reboot the Salt Master node.

   Caution

   All hardware nodes must be rebooted after executing the commands below. If the nodes do not reboot for a long time, execute the below commands again or reboot the nodes manually.

   Verify that you have a possibility to log in to nodes through IPMI in case of emergency.

   1. For KVM nodes:

      salt --async -C 'I@salt:control' cmd.run 'salt-call state.sls \
      linux.system.repo,linux.system.user,openssh,linux.network;reboot'
      

   2. For compute nodes:

      salt --async -C 'I@nova:compute' pkg.install bridge-utils,vlan
      

      salt --async -C 'I@nova:compute' cmd.run 'salt-call state.sls \
      linux.system.repo,linux.system.user,openssh,linux.network;reboot'
      

   3. For gateway nodes, execute the following command only for the deployments with OVS setup with physical gateway nodes:

      salt --async -C 'I@neutron:gateway' cmd.run 'salt-call state.sls \
      linux.system.repo,linux.system.user,openssh,linux.network;reboot'
      

   The targeted KVM, compute, and gateway nodes will stop responding after a couple of minutes. Wait until all of the nodes reboot.

7. Verify that the targeted nodes are up and running:

   salt -C 'I@salt:control or I@nova:compute or I@neutron:gateway or I@ceph:osd' \
   test.ping
   

8. Check the previously created trigger file to verify that the targeted nodes are actually rebooted:

   salt -C 'I@salt:control or I@nova:compute or I@neutron:gateway' \
   cmd.run 'if [ -f "/run/is_rebooted" ];then echo "Has not been rebooted!";else echo "Rebooted";fi'
   

   All nodes should be in the Rebooted state.

9. Verify that the hardware nodes have the required network configuration. For example, verify the output of the ip a command:

   salt -C 'I@salt:control or I@nova:compute or I@neutron:gateway or I@ceph:osd' \
   cmd.run "ip a"
   

Deploy VCP¶

The virtualized control plane (VCP) is hosted by KVM nodes deployed by MAAS. Depending on the cluster type, the VCP runs Kubernetes or OpenStack services, database (MySQL), message queue (RabbitMQ), Contrail, and support services, such as monitoring, log aggregation, and a time-series metric database. VMs can be added to or removed from the VCP allowing for easy scaling of your MCP cluster.

After the KVM nodes are deployed, Salt is used to configure Linux networking, appropriate repositories, host name, and so on by running the linux Salt state against these nodes. The libvirt packages configuration, in its turn, is managed by running the libvirt Salt state.

root@prx01:# lsblk /dev/vda
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                   252:0    0   20G  0 disk
├─vda2                252:2    0 1002M  0 part /boot
├─vda3                252:3    0   19G  0 part
│ ├─vg0-home          253:1    0  100M  0 lvm  /home
│ ├─vg0-var_tmp       253:4    0  500M  0 lvm  /var/tmp
│ ├─vg0-tmp           253:2    0  500M  0 lvm  /tmp
│ ├─vg0-root          253:0    0  9.5G  0 lvm  /
│ ├─vg0-var_log_audit 253:5    0  500M  0 lvm  /var/log/audit
│ └─vg0-var_log       253:3    0  2.9G  0 lvm  /var/log
└─vda1                252:1    0    1M  0 part

root@cfg01:~# salt kvm01* pillar.items salt:control:cluster:internal:node:prx01:size --out json
{
    "kvm01.<CLUSTER_NAME>.local": {
        "salt:control:cluster:internal:node:prx01:size": "openstack.proxy"
    }
}

root@cfg01:~# salt kvm01* pillar.items salt:control:size:openstack.control --out json
{
    "kvm01.<CLUSTER_NAME>.local": {
        "salt:control:size:openstack.control": {
            "net_profile": "default",
            "ram": 32768,
            "cpu": 8,
            "disk_profile": "small"
        }
    }
}

root@cfg01:~# salt kvm01* pillar.items virt:disk --out json
{
    "kvm01.<CLUSTER_NAME>.local": {
        "virt:disk": {
            "small": [
                {
                    "system": {
                        "size": 8000
                    }
                }
            ]
        }
    }
}

salt:
  control:
    enabled: true
    virt_enabled: true
    cluster:
      mycluster:
        domain: neco.virt.domain.com
        engine: virt
        # Cluster global settings
        seed: cloud-init
        cloud_init:
          user_data:
            disable_ec2_metadata: true
            resize_rootfs: True
            timezone: UTC
            ssh_deletekeys: True
            ssh_genkeytypes: ['rsa', 'dsa', 'ecdsa']
            ssh_svcname: ssh
            locale: en_US.UTF-8
            disable_root: true
            apt_preserve_sources_list: false
            apt:
              sources_list: ""
              sources:
                ubuntu.list:
                  source: ${linux:system:repo:ubuntu:source}
                mcp_saltstack.list:
                  source: ${linux:system:repo:mcp_saltstack:source}

salt:
 control:
   cluster:
     mycluster:
       node:
         ubuntu1:
           provider: node01.domain.com
           image: ubuntu.qcow
           size: medium
           cloud_init:
             network_data:
               networks:
               - <<: *private-ipv4
                 ip_address: 192.168.0.161

Deploy CI/CD¶

The automated deployment of the MCP components is performed through CI/CD that is a part of MCP DriveTrain along with SaltStack and Reclass. CI/CD, in its turn, includes Jenkins, Gerrit, and MCP Registry components. This section explains how to deploy a CI/CD infrastructure.

To deploy CI/CD automatically:

1. Deploy a customer-specific CI/CD using Jenkins as part of, for example, an OpenStack cloud environment deployment:

   1. Log in to the Jenkins web UI available at salt_master_management_address:8081 with the following credentials:
      • Username: admin
      • Password: r00tme
   2. Use the Deploy - OpenStack pipeline to deploy cicd cluster nodes as described in Deploy an OpenStack environment. Start with Step 7 in case of the online deployment and with Step 8 in case of the offline deployment.

2. Once the cloud environment is deployed, verify that the cicd cluster is up and running.

3. Disable the Jenkins service on the Salt Master node:

   • For the MCP versions 2018.11.0 and below:

     systemctl stop jenkins
     systemctl disable jenkins
     

   • For the MCP versions 2019.2.0 and newer, add following pillars to infra/config/jenkins.yml:

     parameters:
       docker:
         client:
           stack:
             jenkins:
               service:
                 master:
                   deploy:
                     replicas: 0
                 slave01:
                   deploy:
                     replicas: 0
     

4. Skip the jenkins.client state on the Salt Master node by adding the following pillars to infra/config/jenkins.yml:

   parameters:
     jenkins:
       client:
         enabled: false
   

5. Refresh pillars on the Salt Master node:

   salt-call saltutil.clear_cache && salt-call saltutil.refresh_pillar
   

6. For the MCP versions 2019.2.0 and newer, update the Jenkins service configuration in Docker on the Salt Master node:

   salt-call state.apply docker.client
   

Deploy an OpenStack environment¶

This section explains how to configure and launch the OpenStack environment deployment pipeline. This job is run by Jenkins through the Salt API on the functioning Salt Master node and deployed hardware servers to set up your MCP OpenStack environment.

Run this Jenkins pipeline after you configure the basic infrastructure as described in Deploy MCP DriveTrain. Also, verify that you have successfully applied the linux and salt states to all physical and virtual nodes for them not to be disconnected during network and Salt Minion setup.

To automatically deploy an OpenStack environment:

1. Log in to the Salt Master node.

2. For the OpenContrail setup, add the version-specific parameters to the <cluster_name>/opencontrail/init.yml file of your Reclass model. For example:

   parameters:
     _param:
       opencontrail_version: 4.1
       linux_repo_contrail_component: oc41
   

3. Set up network interfaces and the SSH access on all compute nodes:

   salt -C 'I@nova:compute' cmd.run 'salt-call state.sls \
      linux.system.user,openssh,linux.network;reboot'
   

4. If you run OVS, run the same command on physical gateway nodes as well:

   salt -C 'I@neutron:gateway' cmd.run 'salt-call state.sls \
      linux.system.user,openssh,linux.network;reboot'
   

5. Verify that all nodes are ready for deployment:

   salt '*' state.sls linux,ntp,openssh,salt.minion
   

   Caution

   If any of these states fails, fix the issue provided in the output and re-apply the state before you proceed to the next step. Otherwise, the Jenkins pipeline will fail.

6. In a web browser, open http://<ip address>:8081 to access the Jenkins web UI.

   Note

   The IP address is defined in the classes/cluster/<cluster_name>/cicd/init.yml file of the Reclass model under the cicd_control_address parameter variable.

7. Log in to the Jenkins web UI as admin.

   Note

   To obtain the password for the admin user, run the salt "cid*" pillar.data _param:jenkins_admin_password command from the Salt Master node.

8. In the global view, verify that the git-mirror-downstream-mk-pipelines and git-mirror-downstream-pipeline-library pipelines have successfully mirrored all content.

9. Find the Deploy - OpenStack job in the global view.

10. Select the Build with Parameters option from the drop-down menu of the Deploy - OpenStack job.

11. Specify the following parameters:

    Deploy - OpenStack environment parameters¶

    Parameter	Description and values
    ASK_ON_ERROR	If checked, Jenkins will ask either to stop a pipeline or continue execution in case of Salt state fails on any task
    STACK_INSTALL	Specifies the components you need to install. The available values include: core kvm cicd openstack ovs or contrail depending on the network plugin. ceph stacklight oss Note For the details regarding StackLight LMA (stacklight) with the DevOps Portal (oss) deployment, see Deploy StackLight LMA with the DevOps Portal.
    BATCH_SIZE Added since 2019.2.6 update	The batch size for Salt commands targeted for a large amount of nodes. Disabled by default. Set to an absolute number of nodes (integer) or percentage, for example, 20 or 20%. For details, see Configure Salt Master threads and batching.
    DIST_UPGRADE_NODES Added since 2019.2.8 update	Select to run apt-get dist-upgrade on all cluster nodes before deployment. Disabled by default.
    SALT_MASTER_CREDENTIALS	Specifies credentials to Salt API stored in Jenkins, included by default. See View credentials details used in Jenkins pipelines for details.
    SALT_MASTER_URL	Specifies the reachable IP address of the Salt Master node and port on which Salt API listens. For example, http://172.18.170.28:6969 To find out on which port Salt API listens: Log in to the Salt Master node. Search for the port in the /etc/salt/master.d/_api.conf file. Verify that the Salt Master node is listening on that port: netstat -tunelp | grep <PORT>
    STACK_TYPE	Specifies the environment type. Use physical for a bare metal deployment

12. Click Build.

13. Once done, configure the Salt Master node password expiration as described in Modify Salt Master password expiration.

Deploy a multi-site OpenStack environment¶

MCP DriveTrain enables you to deploy several OpenStack environments at the same time.

To deploy a multi-site OpenStack environment, repeat the Deploy an OpenStack environment procedure as many times as you need specifying different values for the SALT_MASTER_URL parameter.

Deploy a Kubernetes cluster¶

The MCP Containers as a Service architecture enables you to easily deploy a Kubernetes cluster on bare metal with Calico plugin set for Kubernetes networking.

This section explains how to configure and launch the Kubernetes cluster deployment pipeline using DriveTrain.

You can enable an external Ceph RBD storage in your Kubernetes cluster as required. For new deployments, enable the corresponding parameters while creating your deployment metadata model as described in Create a deployment metadata model. For existing deployments, follow the Enable an external Ceph RBD storage procedure.

You can also deploy ExternalDNS to set up a DNS management server in order to control DNS records dynamically through Kubernetes resources and make Kubernetes resources discoverable through public DNS servers.

If you have a predeployed OpenStack environment, you can deploy a Kubernetes cluster on top of OpenStack and enable the OpenStack cloud provider functionality. It allows you to leverage Cinder volumes and Neutron LBaaS (Octavia) that enhance the Kubernetes cluster functionality.

^{Added in 2019.2.3} If you want to enable Helm for managing Kubernetes charts, configure your deployment model as described in Enable Helm support. Once configured, Helm will be deployed on the Kubernetes cluster using the corresponding DriveTrain pipeline.

Depending on your cluster configuration, proceed with one of the sections listed below.