MCP Operations Guide¶

Reinstall MAAS¶

If your MAAS instance is lost or broken, you can reinstall it. This section describes how to install MAAS from the Ubuntu Server 16.04 qcow2 image.

To reinstall MAAS:

1. Create a cloud-config disk:

   1. For example, create a configuration file named config-drive.yaml:

      #cloud-config
      debug: True
      ssh_pwauth: True
      disable_root: false
      chpasswd:
        list: |
          root:r00tme
          ubuntu:r00tme
        expire: False
      

      Note

      You must change the default password.

   2. Create the configuration drive:

      export VM_CONFIG_DISK="/var/lib/libvirt/images/maas/maas-config.iso"
      cloud-localds  --hostname maas01 --dsmode local ${VM_CONFIG_DISK}  config-drive.yaml
      

2. Create a VM system disk using the preloaded qcow2 image. For example:

   wget http://cloud-images.ubuntu.com/releases/xenial/release-20180306/ubuntu-16.04-server-cloudimg-amd64-disk1.img -O \
   /var/lib/libvirt/images/maas/maas-system-backend.qcow2
   
   export VM_SOURCE_DISK="/var/lib/libvirt/images/maas/maas-system.qcow2"
   qemu-img create -b /var/lib/libvirt/images/maas/maas-system-backend.qcow2 -f qcow2 ${VM_SOURCE_DISK} 100G
   

3. Create a VM using the predefine-vm script. For example:

   export MCP_VERSION="master"
   wget https://github.com/Mirantis/mcp-common-scripts/blob/${MCP_VERSION}/predefine-vm/define-cfg01-vm.sh
   
   chmod 0755 define-vm.sh
   export VM_NAME="maas01.[CLUSTER_DOMAIN]"
   

   Note

   You may add other optional variables that have default values and set them according to your deployment configuration. These variables include:

   • VM_MGM_BRIDGE_NAME="br-mgm"
   • VM_CTL_BRIDGE_NAME="br-ctl"
   • VM_MEM_KB="8388608"
   • VM_CPUS="4"

   
   

   The br-mgm and br-ctl values are the names of the Linux bridges. See MCP Deployment Guide: Prerequisites to deploying MCP DriveTrain for details. Custom names can be passed to a VM definition using the VM_MGM_BRIDGE_NAME and VM_CTL_BRIDGE_NAME variables accordingly.

4. Boot the created VM.

5. Log in to the VM using the previously defined password.

6. Proceed with installing MAAS:

   sudo apt-get install maas
   

7. Configure MAAS as required to complete the installation.

8. Verify the installation by opening the MAAS web UI:

   http://<MAAS-IP-ADDRESS>:5240/MAAS
   

9. If you have installed MAAS from the packages, create an initial (administrative) user first to log in to the MAAS web UI:

   sudo maas createadmin --username=<PROFILE> --email=<EMAIL_ADDRESS>
   

Add an SSH key¶

To simplify access to provisioned nodes, add an SSH key that MAAS will deliver when provisioning these nodes.

To add an SSH key:

1. In the MAAS web UI, open the user page by clicking on your user name at the top-right conner.
2. Find the SSH keys section.
3. From the Source drop-down menu, specify the source of an SSH key using one of the options:
   • Launchpad, specifying your Launchpad ID
   • Github, specifying your Github ID
   • Upload, placing an existing public key to the Public key edit box
4. Click Import.

Add a boot image¶

You can select images with appropriate CPU architectures that MAAS will import, regularly sync, and deploy to the managed nodes.

To add a new boot image:

1. In the MAAS web UI, open the Images page.
2. Select the releases you want to make available, as well as any architecture.
3. Click Save selection.

Add a subnet¶

You can add new networking elements in MAAS such as fabrics, VLANs, subnets, and spaces. MAAS should detect new network elements automatically. Otherwise, you can add them manually.

To add a new subnet:

1. In the MAAS web UI, open the Subnets page.
2. Select Subnet in the Add drop-down menu at the top-right corner.
3. Specify Name, CIDR, Gateway IP, DNS servers, Fabric & VLAN, and Space.
4. Click Add subnet.

Enable DHCP on a VLAN¶

Before enabling DHCP, ensure that you have the MAAS node network interface properly configured to listen to VLAN.

You can use external DHCP or enable MAAS-managed DHCP. Using an external DHCP server for enlistment and commissioning may work but is not supported. High availability also depends upon MAAS-managed DHCP.

To enable MAAS-managed DHCP on a VLAN:

1. In the MAAS web UI, open the Subnets page.
2. Click on the VLAN you want to enable DHCP on.
3. In the VLAN configuration panel, you find DHCP Disabled.
4. From the Take action drop-down menu at the top-right corner, select the Provide dhcp item.
5. In the Provide DHCP panel, verify or change the settings for Rack controller, Subnet, Dynamic range start IP, Dynamic range end IP.
6. Click Provide dhcp.
7. In the VLAN configuration panel, verify that DHCP is enabled.

Enable device discovery¶

MAAS provides passive and active methods of device discovery.

Passive methods include:

• Listening to ARP requests
• DNS advertisements

To enable passive device discovery:

1. In the MAAS web UI, open the MAAS dashboard.
2. On the MAAS dashboard, turn on the Discovery enabled switch.
3. Verify if you can see the discovered devices on the MAAS dashboard.

Active methods include active subnet mapping that forces MAAS to discover nodes on all the subnets with enabled active mapping using an active subnet mapping interval value.

To enable active subnet mapping:

1. In the MAAS web UI, open the Settings page.
2. Go to the Device Discovery section.
3. From the drop-down menu, select the value for Active subnet mapping interval.
4. Open the Subnets page.
5. Click the subnet you want to enable active mapping on.
6. In the Subnet summary section, turn on the Active mapping switch.

Add a new node¶

Using MAAS, you can add new nodes in an unattended way called enlistment or manually, when enlistment does not work.

MAAS enlistment uses a combination of DHCP with TFTP and PXE technologies.

To add a new node manually:

1. In the MAAS web UI, open the Nodes page.

2. Click the Add hardware drop-down menu at the top-right corner and select Add machine.

3. Set Machine name, Domain, Architecture, Minimum Kernel, Zone, MAC Address, Power type.

   Note

   See Configure power management for more details on power types.

4. Click Save machine.

MAAS will add the new machine to the list of nodes with the status Comissioning.

Configure power management¶

MAAS supports many types of power control, from standard IPMI to non-standard types such as virsh, VMware, Nova, or even completely manual ones that require operator intervention. While most servers may use their own custom vendor management, for example, iLO or DRAC, standard IPMI controls are also supported, and you can use IPMI as shown in the following example.

To configure IPMI node power management type:

1. In the MAAS web UI, open the Nodes page.
2. In the list of nodes, select the one you want to configure.
3. In the machine configuration window, go to the Power section and click the Edit button at the top-right conner.
4. Select IPMI for Power type from the drop-down menu.
5. Specify parameters for the IPMI power type: Power driver, IP address, Power user, Power password, and Power MAC.
6. Click Save changes.

After saving the changes, MAAS will verify that it can manage the node through IPMI.

Commission a new node¶

When you add a new node, MAAS automatically starts commissioning the node once configuration is done. Also, you can commission a node manually.

To commision a new node manually:

1. In the MAAS web UI, open the Nodes page.
2. In the list of nodes, click the node you want to configure.
3. In the node configuration window, click Take action at the top-right conner and select Commission.
4. Select additional options by setting the appropriate check boxes to allow SSH access and prevent machine from powering off, retain network and storage configuration if you want to preserve data on the node. For example, if a node comes from an existing cloud with an instance on it.
5. Click Go to start commissioning.
6. Verify that the node status has changed to Ready and hardware summary in the node configuration window has been filled with values other than zeros, which means commisionining was successful.

Deploy a node¶

Once a node has been commissioned, you can deploy it.

The deployment operation includes installing an operating system and copying the SSH keys imported to MAAS. As a result, you can access the deployed node through SSH using the default user account ubuntu.

To deploy a node:

1. In the MAAS web UI, open the Nodes page.
2. In the list of nodes, verify that the commisioned node is in the Ready state.
3. Click on the node to open the configuration window.
4. In the node configuration window, click the Take action button at the top-right conner and select the Deploy item.
5. Specify the OS, release, and kernel options.
6. Click Go to start the deployment.

Once the deployment is finished, MAAS will change the node status to Deployed.

Redeploy a node¶

To redeploy a node:

1. In the MAAS web UI, open the Nodes page.
2. In the list of nodes, select the node you want to redeploy.
3. In the node configuration window, click Take action at the top-right coner and select Release.
4. Verify that the node status has changed to Ready.
5. Redeploy the node as described in Deploy a node.

Delete a node¶

To delete a node:

1. In the MAAS web UI, open the Nodes page.
2. In the list of nodes, select the node you want to delete.
3. In the node configuration window, click the Take action button at the top-right coner and select Delete.
4. Click Go to confirm the deletion.

Salt Minion operations¶

salt '[node]' cmd.run '[cmd]'

salt '<NODE_NAME>' service.get_all

salt '<NODE_NAME>' service.restart <SERVICE_NAME>

salt-key

Accepted Keys:
<NODE_NAME>.domain.local
… [snip] ...
Execute salt-key:
Denied Keys:
Unaccepted Keys:
Rejected Keys:

salt '<NODE_NAME>.domain.local' test.ping

<NODE_NAME>
 True

Salt States operations¶

Salt State is a declarative or imperative representation of a system state.

salt-call state.show_top

local:
 ----------
 base:
     - linux
     - ntp
     - salt
     - heka
     - openssh
     - nova
     - opencontrail
     - ceilometer

salt '<NODE_NAME>' state.sls <STATE_NAME>

Salt Formula operations¶

Salt Formula is a declarative or imperative representation of a system configuration.

salt '*' state.show_sls <SLS_FILE_NAME>

salt '*' state.apply test

salt '*' state.apply

salt '*' state.highstate

salt '*' state.apply <SLS_FILE_1>,<SLS_FILE_2>

salt --batch-size 10% -N computes1 state.apply
salt -b 5 -N compute:compute1 state.apply

Replace the Salt Master keys¶

In case your Salt Master keys have been compromised, you can replace both Salt Master CA and RSA SSH keys. The replacement procedure of the Salt Master keys does not affect your cloud environment, only the Salt structure is updated.

Job configuration history¶

The DriveTrain Jenkins provides the capability to inspect the history of jobs configuration changes using the Job Configuration History plugin. This plugin captures and stores the changes to all jobs configured in Jenkins and enables the DriveTrain administrator to view these changes. It allows identifying the date of the change, the user that created the change, and the content of the change itself.

To use the Job Configuration History plugin:

1. Log in to the Jenkins web UI as an administrator using the FQDN of your cloud endpoint and port 8081. For example, https://cloud.example.com:8081.
2. Navigate to Job Config History > Show job configs only.
3. Click the required job and review the list of recorded changes.

Alternatively, you can access the history of a job by clicking Job Config History from the particular job view itself.

Abort a hung build in Jenkins¶

This section provides the instruction on how to abort the hung Jenkins build if it does not restore after the Jenkins dedicated node is restared, for example.

To abort a hung build, select from the following options

• Abort the job build from the Jenkins web UI:

  1. Log in to the Jenkins web UI as an Administrator using the FQDN of your cloud endpoint and the 8081 port. For example, https://cloud.example.com:8081.

  2. Navigate to Manage Jenkins > Script Console.

  3. Run the following script setting the job name and number of the hung build accordingly:

     def build = Jenkins.instance.getItemByFullName("jobName").getBuildByNumber(jobNumber)
     build.doStop()
     build.doKill()
     

• Abort the job build from a cid node (if the previous option did not help):

  1. Log in to any cid node.

  2. Run:

     cd /srv/volumes/jenkins/jobs/<job-name>/builds/
     rm -rf <hung-build-number>
     

  3. Log in to the Jenkins web UI as an Administrator using the FQDN of your cloud endpoint and the 8081 port. For example, https://cloud.example.com:8081.

  4. Navigate to Manage Jenkins > Reload Configuration from Disk, click to reload. Or restart the Jenkins instance.

Enable Jenkins audit logging¶

This section instructs you on how to enable the audit logging in Jenkins by enabling the Audit Trail Jenkins plugin. The plugin allows keeping a log of the users who performed particular Jenkins operations, such as managing and using jobs.

To setup Audit logging in Jenkins:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. In the cicd/control/leader.yml file, configure any of three logger types that include console, file, and syslog.

   Note

   By default, only the console output is collected by Fluentd if enabled.

   Pillars examples:

   1. For the console logger:

      parameters:
        jenkins:
          client:
            audittrail:
              loggers:
                console_logger:
                  type: console
                  output: STD_OUT
                  date_format: "yyyy-MM-dd HH:mm:ss:SSS"
                  log_prefix: ""
      

      Note

      The date_format and log_prefix parameters in the example above are defaults and can be skipped.

   2. For the file logger:

      parameters:
        jenkins:
          client:
            audittrail:
              loggers:
                file_logger:
                  type: file
                  log: /var/jenkins_home/file_logger.log
                  limit: 100
                  count: 10
      

      Note

      The limit parameter stands for the file limit size in MB. The count parameter stands for the number files to keep.

   3. For the syslog logger:

      parameters:
        jenkins:
          client:
            audittrail:
              loggers:
                syslog_logger:
                  type: syslog
                  syslog_server_hostname: 'syslog.host.org'
                  syslog_server_port: 514
                  syslog_facility: SYSLOG
                  app_name: jenkins
                  message_hostname: ""
                  message_format: RFC_3164
      

4. To configure the audit Logging for Jenkins on the Salt Master node, add the similar pillars to infra/config/jenkins.yml.

5. Refresh pillars:

   salt -C 'I@jenkins:client' saltutil.refresh_pillar
   

6. Apply the changes:

   salt -C 'I@jenkins:client:audittrail' state.apply jenkins.client.audittrail
   

Jenkins Matrix-based security authorization¶

The DriveTrain Jenkins uses Matrix-based security authorization by default. It allows you to grant specific permissions to users and groups. Jenkins uses DriveTrain OpenLDAP server as an identity provider and authentication server.

By default, the Jenkins server includes the following user groups:

• admins

  Contains administrative users with the Jenkins Administer permission.

• Authenticated Users

  Includes all users authenticated through the DriveTrain OpenLDAP server. This group has no permissions configured by default.

The Matrix-based security plugin enables the operator to configure the following types of permissions:

• Overall

  Either Administer or Read permissions can be set overall.

• Credentials

  Permissions to create, delete, update, and view authentication credentials, and manage domains.

• Gerrit

  Permissions to manually trigger and retrigger Gerrit integration plugin to run specific jobs normally initiated by the plugin.

• Agents

  Permissions to manage Jenkins agents on worker nodes.

• Job

  Permissions for specific operations on Jenkins jobs, including build creation, configuration, and execution.

• Run

  Permissions to run and rerun jobs.

• View

  Permissions to manage views in the Jenkins UI.

• SCM

  Permissions to use SCM tags.

• Metrics

  Permissions to view and configure metrics.

• Lockable resources

  Permissions to reserve and unlock lockable resources manually.

• Artifactory

  Permissions to use the Artifactory integration plugin (only if Artifactory is installed).

To configure the Matrix-based Security Authorization:

1. Log in to the Jenkins web UI as an administrator using the FQDN of your cloud endpoint and port 8081. For example, https://cloud.example.com:8081.
2. Navigate to Manage Jenkins > Configure Global Security.
3. Scroll to the Authorization section to view and change the Matrix-based security settings.

Remove executors on Jenkins master¶

The DriveTrain Jenkins enabled on the Salt Master node enables you to run any job on any Jenkins slave or master. Though, running a job on Jenkins master can lead to job failures since Jenkins master is not intended to be a job executor.

Starting from the MCP 2019.2.4 maintenance update, MCP disables executors on Jenkins master by default to prevent failures of the jobs that run on the Salt Master node. For the MCP versions earlier than 2019.2.4, Mirantis recommends setting zero executors on Jenkins master to disable jobs scheduling on this agent node as described below.

To set zero executors on Jenkins master on the Salt Master node:

1. Log in to the Salt Master node.

2. In ./classes/cluster/<cluster_name>/infra/config/jenkins.yml, add the following pillar data:

   parameters:
     _param:
       jenkins:
         client:
           ...
           node:
             master:
               num_executors: 0
              ...
   

3. Refresh pillars:

   salt -C 'I@salt:master' saltutil.refresh_pillar
   

4. Apply the changes:

   salt -C 'I@salt:master' state.apply jenkins.client
   

Remove anonymous access to Jenkins on the Salt Master node¶

The DriveTrain Jenkins enabled on the Salt Master node is configured to allow anonymous users to access the Jenkins web UI including the listing of the Jenkins jobs and builds in the web UI.

For security reasons, starting from the MCP 2019.2.4 maintenance update, by default, only authorized users have access to Jenkins on the Salt Master node. For the MCP versions earlier than 2019.2.4, Mirantis recommends configuring Jenkins as described below.

To remove anonymous access to Jenkins on the Salt Master node:

1. Log in to the Salt Master node.

2. In ./classes/cluster/<cluster_name>/infra/config/jenkins.yml, replace anonymous with authenticated for jenkins_security_matrix_read:

   parameters:
     _param:
       jenkins_security_matrix_read:
       - authenticated
   

3. Refresh pillars:

   salt -C 'I@salt:master' saltutil.refresh_pillar
   

4. Apply the changes:

   salt -C 'I@salt:master' state.apply jenkins.client
   

Use SSH Jenkins slaves¶

By default, Jenkins uses Java Network Launch Protocol (JNLP) for Jenkins slave connection. Starting from the MCP 2019.2.5 maintenance update, you can set up SSH connection for Jenkins slaves instead of JNLP using the steps below.

To use SSH connection instead of JNLP for Jenkins slaves:

1. Log in to the Salt Master node.

2. Configure Jenkins Master for the Salt Master node to use SSH Jenkins slaves:

   1. Verify your existing SSH keys for Jenkins admin key:

      salt-call pillar.get _param:jenkins_admin_public_key_generated
      salt-call pillar.get _param:jenkins_admin_private_key_generated
      

      The system output must be not empty.

      If you do not have SSH keys, generate ones:

      ssh-keygen
      

   2. In ./classes/cluster/<cluster_name>/infra/config/jenkins.yml:

      1. Replace the system.docker.swarm.stack.jenkins.slave_single or system.docker.swarm.stack.jenkins.jnlp_slave_single class (the one that is present in model) with the following class:

         classes:
         ...
         - system.docker.swarm.stack.jenkins.ssh_slave_single
         

      2. Remove the following classes if present:

         classes:
         ...
         - system.docker.client.images.jenkins_master
         - system.docker.client.images.jenkins_slave
         

      3. Change the Jenkins slave type to ssh instead of jnlp:

         parameters:
           ...
           jenkins:
             client:
               node:
                 slave01:
                   launcher:
                     type: ssh
         

      4. Add the SSH keys parameters to the parameters section:

         • If you use existing SSH keys:

           parameters:
             _param:
               ...
               jenkins_admin_public_key: ${_param:jenkins_admin_public_key_generated}
               jenkins_admin_private_key: ${_param:jenkins_admin_private_key_generated}
               ...
           

         • If you generated new SSH keys in the step 2.1:

           parameters:
             _param:
               ...
               jenkins_admin_public_key: <ssh-public-key>
               jenkins_admin_private_key: <ssh-private-key>
               ...
           

3. Remove the JNLP slave from Jenkins on Salt Master node:

   1. Log in to Salt Master node Jenkins web UI.
   2. Navigate to Manage Jenkins > Manage nodes.
   3. Select slave01 > Delete agent. Click yes to confirm.
   
   

4. Configure Jenkins Master for the cid nodes to use SSH Jenkins slaves:

   1. Verify that the Jenkins SSH key is defined in the Reclass model:

      salt 'cid01*' pillar.get _param:jenkins_admin_public_key
      salt 'cid01*' pillar.get _param:jenkins_admin_private_key
      

   2. In ./classes/cluster/<cluster_name>/cicd/control/leader.yml:

      1. Replace the system.docker.swarm.stack.jenkins class with system.docker.swarm.stack.jenkins.master and the system.docker.swarm.stack.jenkins.jnlp_slave_multi class with system.docker.swarm.stack.jenkins.ssh_slave_multi if present, or add system.docker.swarm.stack.jenkins.ssh_slave_multi explicitly.

      2. Add the system.jenkins.client.ssh_node class right below the system.jenkins.client.node class:

         classes:
         ...
         - system.jenkins.client.node
         - system.jenkins.client.ssh_node
         

5. Remove the JNLP slaves from Jenkins on the cid nodes:

   1. Log in to cid Jenkins web UI.
   2. Navigate to Manage Jenkins > Manage nodes.
   3. Delete slave01, slave02 and slave03 using the menu. For example: slave01 > Delete agent. Click yes to confirm.
   
   

6. Refresh pillars:

   salt -C 'I@jenkins:client' saltutil.refresh_pillar
   salt -C 'I@docker:client' saltutil.refresh_pillar
   

7. Pull the ssh-slave Docker image:

   salt -C 'I@docker:client:images' state.apply docker.client.images
   

8. Apply the changes:

   salt -C 'I@jenkins:client and I@docker:client' state.apply docker.client
   salt -C 'I@jenkins:client' state.apply jenkins.client
   

Enable HTTPS access from Jenkins to Gerrit¶

By default, Jenkins uses the SSH connection to access Gerrit repositories. This section explains how to set up the HTTPS connection from Jenkins to Gerrit repositories.

To enable access from Jenkins to Gerrit through HTTPS:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. In the cicd/control/leader.yml file:

   1. Replace the system.jenkins.client.credential.gerrit class with the system.jenkins.client.credential.gerrit_http class:

      classes:
      ...
      - system.jenkins.client.credential.gerrit_http
      

   2. Redefine the jenkins_gerrit_url parameter as follows:

      jenkins_gerrit_url: "https://${_param:haproxy_gerrit_bind_host}:${_param:haproxy_gerrit_bind_port}"
      

4. Refresh pillars:

   salt -C 'I@jenkins:client and not I@salt:master' saltutil.refresh_pillar
   

5. Apply the changes:

   salt -C 'I@jenkins:client and not I@salt:master' state.apply jenkins.client
   

Manage secrets in the Reclass model¶

MCP uses the GPG encryption to protect sensitive data in the Git repositories of the Reclass model. The private key from the encrypted data is stored on the Salt Master node and is available to the root user only. Usually, the data stored in the secrets.yml files located in the /srv/salt/reclass/cluster directory is encrypted. The decryption key is located in a keyring in /etc/salt/gpgkeys.

The secrets encryption feature is not enabled by default. To enable the feature, define secrets_encryption_enabled: 'True' in the Cookiecutter context before the deployment. See MCP Deployment Guide: Infrastructure related parameters: Salt Master for the details.

To change a password:

1. Get the ID of the private key in question:

   # GNUPGHOME=/etc/salt/gpgkeys gpg --list-secret-keys
   

   The machine-readable version of the above command:

   # GNUPGHOME=/etc/salt/gpgkeys gpg --list-secret-keys --with-colons | awk -F: -e '/^sec/{print $5}'
   

2. Encrypt the new password:

   # echo -ne <new_password> | GNUPGHOME=/etc/salt/gpgkeys gpg --encrypt --always-trust -a -r <key_id>
   

3. Add the new password to secrets.yml.

To decrypt the data:

To get the decoded value, pass the encrypted value to the command:

# GNUPGHOME=/etc/salt/gpgkeys gpg --decrypt

To change the secret encryption private key:

1. Add a new key to keyring in /etc/salt/gpgkeys using one of the following options:

   • Import the existing key:

     # GNUPGHOME=/etc/salt/gpgkeys gpg --import < <key_file>
     

   • Create a new key:

     # GNUPGHOME=/etc/salt/gpgkeys gpg --gen-key
     

2. Replace all encrypted fields in all secrets.yml files with the encrypted value for new key_id.

Configure allowed and rejected IP addresses for the GlusterFS volumes¶

This section provides the instruction on how to configure the list of allowed and rejected IP addresses for the GlusterFS volumes.

By default, MCP restricts the access to the control network for all preconfigured GlusterFS volumes.

To configure the GlusterFS authentication:

1. Log in to the Salt Master node.

2. Open your project Git repository with the Reclass model on the cluster level.

3. In the infra/glusterfs.yml file, configure the GlusterFS authentication depending on the needs of your MCP deployment:

   • To adjust the list of allowed and rejected IP addresses on all preconfigured GlusterFS volumes, define the glusterfs_allow_ips and glusterfs_reject_ips parameters as required:

     parameters:
       _param:
         glusterfs_allow_ips: <comma-seprated list of IPs>
         glusterfs_reject_ips: <comma-seprated list of IPs>
     

     Note

     You can use the \* wildcard to specify the IP ranges.

     Configuration example:

     parameters:
       _param:
         glusterfs_allow_ips: 10.0.0.1, 192.168.1.*
         glusterfs_reject_ips: 192.168.1.201
     

     The configuration above allows the access to all GlusterFS volumes from 10.0.0.1 and all IP addresses in the 192.168.1.0/24 network except for 192.168.1.201.

   • To change allowed and rejected IP addresses for a single volume:

     parameters:
       glusterfs:
         server:
           volumes:
             <volume_name>:
               options:
                 auth.allow: <comma-seprated_list_of_IPs>
                 auth.reject: <comma-seprated_list_of_IPs>
     

   • To define the same access-control lists (ACL) as for all preconfigured GlusterFS volumes to a custom GlusterFS volume, define the auth.allow and auth.reject options for the targeted volume as follows:

     auth.allow: ${_param:glusterfs_allow_ips}
     auth.reject: ${_param:glusterfs_reject_ips}
     

4. Apply the changes:

   salt -I 'glusterfs:server:role:primary' state.apply glusterfs
   

Manage users in OpenLDAP¶

DriveTrain uses OpenLDAP to provide authentication and metadata for MCP users. This section describes how to create a new user entry in the OpenLDAP service through the Reclass cluster metadata model and grant the user permissions to access Gerrit and Jenkins.

To add a user to an OpenLDAP server:

1. Log in to the Salt Master node.

2. Check out the latest version of the Reclass cluster metadata model from the Git repository for your project.

3. Create a new directory called people in classes/cluster/<CLUSTER_NAME>/cicd/:

   mkdir classes/cluster/<cluster_name>/cicd/people
   

   New user definitions will be added to this directory.

4. Create a new YAML file in the people directory for a new user. For example, joey.yml:

   touch classes/cluster/<cluster_name>/cicd/people/joey.yml
   

5. In the newly created file, add the user definition. For example:

   parameters:
     _param:
       openldap_pw_joey: "<ENCRYPTED_PASSWORD>"
     openldap:
       client:
         entry:
           people:
             entry:
               jdoe:
                 attr:
                   uid: joey
                   userPassword: ${_param:openldap_pw_joey}
                   uidNumber: 20600
                   gidNumber: 20001
                   gecos: "Joey Tribbiani"
                   givenName: Joey
                   sn: Tribbiani
                   homeDirectory: /home/joey
                   loginShell: /bin/bash
                   mail: joey@domain.tld
                 classes:
                   - posixAccount
                   - inetOrgPerson
                   - top
                   - shadowAccount
   

   Parameters description:

   • openldap_pw_joey

     The user password for the joey user that can be created using the following example command:

     echo "{CRYPT}$(mkpasswd --rounds 500000 -m sha-512 \
       --salt `head -c 40 /dev/random | base64 | sed -e 's/+/./g' \
       |  cut -b 10-25` 'r00tme')"
     

     Substitute r00tme with a user encrypted password.

   • uid

     The case-sensitive user ID to be used as a login ID for Gerrit, Jenkins, and other integrated services.

   • userPassword: ${_param:openldap_pw_joey}

     The password for the joey user, same as the openldap_pw_joey value.

   • gidNumber

     An integer uniquely identifying a group in an administrative domain, which a user should belong to.

   • uidNumber

     An integer uniquely identifying a user in an administrative domain.

6. Add the new user definition from joey.yml as a class in classes/cluster/<CLUSTER_NAME>/cicd/control/leader.yml:

   classes:
     ...
     - cluster.<CLUSTER_NAME>.cicd.control
     - cluster.<CLUSTER_NAME>.cicd.people.joey
   

   By defining the cluster level parameters of the joey user and including it in the classes section of cluster/<CLUSTER_NANE>/cicd/control/leader.yml, you import the user data to the cid01 node inventory, although the parameter has not been rendered just yet.

7. Commit the change.

8. Update the copy of the model on the Salt Master node:

   sudo git -C /srv/salt/reclass pull
   

9. Synchronize all Salt resources:

   sudo salt '*' saltutil.sync_all
   

10. Apply the changes:

    sudo salt 'cid01*' state.apply openldap
    

    Example output for a successfully created user:

            ID: openldap_client_cn=joey,ou=people,dc=deploy-name,dc=local
      Function: ldap.managed
        Result: True
       Comment: Successfully updated LDAP entries
       Started: 18:12:29.788665
      Duration: 58.193 ms
       Changes:
                ----------
                cn=joey,ou=people,dc=deploy-name,dc=local:
                    ----------
                    new:
                        ----------
                        cn:
                            - joey
                        gecos:
                            - Joey Tribbiani
                        gidNumber:
                            - 20001
                        givenName:
                            - Joey
                        homeDirectory:
                            - /home/joey
                        loginShell:
                            - /bin/bash
                        mail:
                            - joey@domain.tld
                        objectClass:
                            - inetOrgPerson
                            - posixAccount
                            - shadowAccount
                            - top
                        sn:
                            - Tribbiani
                        uid:
                            - joey
                        uidNumber:
                            - 20060
                        userPassword:
                            - {CRYPT}$6$rounds=500000$KaJBYb3F8hYMv.UEHvc0...
                    old:
                        None
    
    Summary for cid01.domain.tld
    ------------
    Succeeded: 7 (changed=1)
    Failed:    0
    ------------
    Total states run:     7
    Total run time: 523.672 ms
    

Disable LDAP authentication on host OS¶

This section describes how to disable LDAP authentication on a host operating system.

To disable LDAP authentication:

1. Open your Git project repository with the Reclass model on the cluster level.

2. In cluster/<cluster_name>/infra/auth/ldap.yml, disable LDAP:

   ldap:
     enabled: false
   

3. Enforce the linux.system update:

   salt '<target_node>*' state.sls linux.system
   

4. Clean up nodes:

   salt '<target_node>*' cmd.run 'export DEBIAN_FRONTEND=noninteractive; apt purge -y libnss-ldapd libpam-ldapd; sed -i "s/ ldap//g" /etc/nsswitch.conf'
   

Enable Gerrit audit logging¶

This section instructs you on how to enable the audit logging in Gerrit by configuring the httpd requests logger. Fluentd collects the files with the error logs automatically.

To set up audit logging in Gerrit:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. In the cicd/control/leader.yml file, add following parameters:

   parameters:
     _param:
       ...
       gerrit_extra_opts: "-Dlog4j.configuration=file:///var/gerrit/review_site/etc/log4j.properties"
       gerrit_http_request_log: 'True'
       ...
     linux:
       system:
         file:
           "/srv/volumes/gerrit/etc/log4j.properties":
             contents:
               - log4j.logger.httpd_log=INFO,httpd_log
               - log4j.appender.httpd_log=org.apache.log4j.ConsoleAppender
               - log4j.appender.httpd_log.layout=com.google.gerrit.pgm.http.jetty.HttpLogLayout
   

4. Refresh pillars:

   salt -C 'I@gerrit:client' saltutil.refresh_pillar
   

5. Create the log4j.properties file:

   salt -C 'I@gerrit:client' state.apply linux.system.file
   

6. Update the Gerrit service:

   salt -C 'I@gerrit:client' state.apply docker.client
   

Configure log rotation using logrotate¶

This section instructs you on how to configure log rotation for selected services using the logrotate utility.

The services that support the log rotation configuration include:

• OpenStack services

  Aodh, Barbican, Ceilometer, Cinder, Designate, Glance, Gnocchi, Heat, Keystone, Neutron, Nova, Octavia, Ironic

• Other services

  atop, Backupninja, Ceph, Elasticsearch, Galera (MySQL), GlusterFS, HAProxy, libvirt, MAAS, MongoDB, NGINX, Open vSwitch, PostgreSQL, RabbitMQ, Redis, Salt, Telegraf

MCP supports configuration of the rotation interval and number of rotations. Configuration of other logrotate options, postrotate and prerotate actions, and so on, are not supported.

To configure log rotation:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. Configure the interval and rotate parameters for the target service as required:

   • logrotate:interval

     Define the rotation time interval. Available values daily, weekly, monthly, and yearly.

   • logrotate:rotate

     Define the number of the rotated logs to be kept. The parameter expects the interger value.

   Use the Logrotate configuration table below to determine where to add the log rotation configuration.

   Logrotate configuration¶

   Service	Pillar path (target)	File path
   Aodh	aodh:server	openstack/telemetry.yml
   atop	linux:system:atop	The root file of the component [7]
   Backupninja	backupninja:client	infra/backup/client_common.yml
   Barbican	barbican:server	openstack/barbican.yml
   Ceilometer server [0]	ceilometer:server	openstack/telemetry.yml
   Ceilometer agent [0]	ceilometer:agent	openstack/compute/init.yml
   Ceph	ceph:common	ceph/common.yml
   Cinder controller [1]	cinder:controller	openstack/control.yml
   Cinder volume [1]	cinder:volume	openstack/control.yml
   Designate	designate:server	openstack/control.yml
   Elasticsearch server	elasticsearch:server	stacklight/log.yml
   Elasticsearch client	elasticsearch:client	stacklight/log.yml
   Galera (MySQL) master	galera:master	openstack/database/master.yml
   Galera (MySQL) slave	galera:slave	openstack/database/slave.yml
   Glance	glance:server	openstack/control.yml
   GlusterFS server [2]	glusterfs:server	The root file of the component [7]
   GlusterFS client [2]	glusterfs:client	The root file of the component [7]
   Gnocchi server	gnocchi:server	openstack/telemetry.yml
   Gnocchi client	gnocchi:client	openstack/control/init.yml
   HAProxy	haproxy:proxy	openstack/proxy.yml
   Heat	heat:server	openstack/control.yml
   Ironic Available since 2019.2.13	ironic:api	openstack/baremetal.yml
   Keystone server	keystone:server	openstack/control.yml
   Keystone client	keystone:client	openstack/control/init.yml
   libvirt	nova:compute:libvirt [3]	openstack/compute/init.yml
   MAAS	maas:region	infra/maas.yml
   MongoDB	mongodb:server	stacklight/server.yml
   Neutron server	neutron:server	openstack/control.yml
   Neutron client	neutron:client	openstack/control/init.yml
   Neutron gateway	neutron:gateway	openstack/gateway.yml
   Neutron compute	neutron:compute	openstack/compute/init.yml
   NGINX	nginx:server	openstack/proxy.yml, stacklight/proxy.yml
   Nova controller	nova:controller	openstack/control.yml
   Nova compute	nova:compute	openstack/compute/init.yml
   Octavia manager [4]	octavia:manager	openstack/octavia_manager.yml
   Octavia client [4]	octavia:client	openstack/control.yml
   Open vSwitch	linux:network:openvswitch	infra/init.yml
   PostgreSQL server	postgresql:server (maas:region) [5]	infra/config/postgresql.yml (infra/maas.yml)
   PostgreSQL client	postgresql:client (maas:region) [5]	infra/config/postgresql.yml (infra/maas.yml)
   RabbitMQ	rabbitmq:server	openstack/message_queue.yml
   Redis	redis:server	openstack/telemetry.yml
   Salt master [6]	salt:master	infra/config/init.yml
   Salt minion [6]	salt:minion	The root file of the component [7]
   Telegraf	telegraf:agent	infra/init.yml, stacklight/server.yml

   [0]	(1, 2) If Ceilometer server and agent are specified on the same node, the server configuration is prioritized.

   [1]	(1, 2) If Cinder controller and volume are specified on the same node, the controller configuration is prioritized.

   [2]	(1, 2) If GlusterFS server and client are specified on the same node, the server configuration is prioritized.

   [3]	Use nova:compute:libvirt as pillar path, but only nova:compute as target.

   [4]	(1, 2) If Octavia manager and client are specified on the same node, the manager configuration is prioritized.

   [5]	(1, 2) PostgreSQL is the dependenсу of MAAS. Configure PostgreSQL from the MAAS pillar only if the service has been installed as a dependency without the postgresql pillar defined. If the postgresql pillar is defined, configure it instead.

   [6]	(1, 2) If the Salt Master and minion are specified on the same node, the master configuration is prioritized.

   [7]	(1, 2, 3, 4) Depending on the nodes where you want to change the configuration, select their components’ root file. For example, infra/init.yml, openstack/control/init.yml, cicd/init.yml, and so on.

   For example, to set log rotation for Aodh to keep logs for the last 4 weeks with the daily rotation interval, add the following configuration to cluster/<cluster_name>/openstack/telemetry.yml:

   parameters:
     aodh:
       server:
         logrotate:
           interval: daily
           rotate: 28
   

4. Apply the logrotate state on the node with the target service:

   salt -C 'I@<target>' saltutil.sync_all
   salt -C 'I@<target>' state.sls logrotate
   

   For example:

   salt -C 'I@aodh:server' state.sls logrotate
   

Configure remote logging for auditd¶

This section instructs you on how to configure remote logging for auditd.

To configure remote logging for auditd:

1. Log in to the Salt Master node.

2. In the classes/cluster/<cluster_name>/ directory, open one of the following files:

   • To configure one remote host for auditd for all nodes, use infra/init.yml.
   • To configure a remote host for a set of nodes, use a specific configuration file. For example, openstack/compute/init.yml for all OpenStack compute nodes.

3. Configure the remote host using the following exemplary pillar:

   parameters:
     audisp:
       enabled: true
       remote:
         remote_server: <ip_address or hostname>
         port: <port>
         local_port: any
         transport: tcp
         ...
         key1: value1
   

4. Refresh pillars on the target nodes:

   salt <nodes> saltutil.refresh_pillar
   

5. Apply the auditd.audisp state on the target nodes:

   salt <nodes> state.apply auditd.audisp
   

Configure memory limits for the Redis server¶

This section instructs you on how to configure the memory rules and limits for the Redis server.

To configure memory limits for the Redis server:

1. Log in to the Salt Master node.

2. In classes/cluster/<cluster_name>/openstack/telemetry.yml, specify the following parameters as required:

   parameters:
     redis:
       server:
         maxmemory: 1073741824 # 1GB
         maxmemory-policy: <memory-policy>
         maxmemory-samples: 3
   

   Supported values for the memory-policy parameter include:

   • volatile-lru - the service removes the key with expiration time set using the Least Recently Used (LRU) algorithm
   • allkeys-lru - the service removes any key according to the LRU algorithm
   • volatile-random - the service removes a random key with an expiration time set
   • allkeys-random - the service removes any random key
   • volatile-ttl - the service removes the key with the nearest expiration time (minor TTL)
   • noeviction - the service does not remove any key but returns an error on write operations

3. Apply the changes:

   salt -C 'I@redis:server' saltutil.refresh_pillar
   salt -C 'I@redis:server' state.apply redis.server
   

Configure multiple NTP servers¶

MCP enables you to configure multiple Network Time Protocol (NTP) servers on new or existing MCP clusters to provide a more flexible and wide NTP support for clustered applications such as Ceph, Galera, and others.

For new MCP clusters, configure multiple NTP servers during the deployment model creation using the ntp_servers parameter passed to Cookiecutter in the following format:

server1.ntp.org,server2.ntp.org,server3.ntp.org

For details, see Networking deployment parameters in MCP Deployment Guide: Create a deployment metadata model.

For existing MCP clusters, configure multiple NTP servers by updating the NTP configuration for MAAS and all nodes of an MCP cluster.

To configure multiple NTP servers for MAAS:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. In infra/maas.yml, update the MAAS pillars using the example below:

   parameters:
     maas:
       region:
         ...
         ntp:
           server1:
             enabled: True
             host: ntp.example1.org
           server2:
             enabled: True
             host: ntp.example2.o
   

4. Update the MAAS configuration:

   salt-call saltutil.refresh_pillar
   salt-call state.apply maas.region
   

To configure multiple NTP servers for all nodes of an MCP cluster:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. In infra/init.yml, update the MAAS pillars using the example below:

   parameters:
     ntp:
       client:
       enabled: true
       stratum:
         primary:
           server: primary.ntp.org
         secondary: # if exist
           server: secondary.ntp.org
         srv_3: # if exist
           server: srv_3.ntp.org
   

4. Update the NTP configuration:

   salt '*' saltutil.refresh_pillar
   salt '*' state.apply ntp.client
   

Configure kernel crash dumping¶

The Kernel Crash Dump (kdump) mechanism allows you to save the system memory events for later analysis. This section instructs you on how to configure kernel crash dumping for all or particular nodes. The dump will be saved in the /var/crash directory. For more details, see Kernel Crash Dump.

To configure kernel crash dumping:

1. Log in to the Salt Master node.

2. Open the cluster level of your deployment model.

3. In <cluster_name>/infra/init.yml, add the following variable:

   parameters:
     linux:
       system:
         kernel_crash_dump:
           enabled: true
   

4. Apply the changes for the required nodes or for all nodes. To apply the changes for all nodes:

   salt -C '*' state.sls linux.system.kernel_crash_dump
   

5. Manually reboot the nodes for which you have enabled kernel crash dumping. For example, to reboot the OpenStack compute nodes:

   salt -C 'cmp1*' system.reboot --async
   

Once done, you can view the dump in the /var/crash directory.

Manage Virtualized Control Plane¶

This section describes operations with the MCP Virtualized Control Plane (VCP).

salt -C "I@salt:control" cmd.run "virsh list | tail -n +3 | awk '{print \$1}' | xargs -I{} virsh dumpxml {} | grep cpu_mode"

Manage compute nodes¶

This section provides instructions on how to manage the compute nodes in your cloud environment.

Manage gateway nodes¶

This section describes how to manage tenant network gateway nodes that provide access to an external network for the environments configured with Neutron OVS as a networking solution.

Manage RabbitMQ¶

A RabbitMQ cluster is sensitive to external factors like network throughput and traffic spikes. When running under high load, it requires special start, stop, and restart procedures.