Apply security updates manually¶

Once you have created an update repository as described in Add an update repository, or if you already have an update repository, you can apply the security updates.

This section instructs you on how to update Ubuntu packages manually. More specifically, the procedure includes update of the KVM nodes one by one, the VMs running on top of each KVM node, and the OpenStack compute nodes.

If you prefer the automated update, use the Apply Ubuntu security updates using Jenkins procedure.

To apply the Ubuntu security updates manually:

1. Log in to the Salt Master node.

2. For OpenStack Ocata with OpenContrail v3.2, update the python-concurrent.futures package. Otherwise, skip this step.

   salt -C "ntw* or nal*" cmd.run "apt-get -y --force-yes upgrade python-concurrent.futures"
   

3. Update the Virtual Control Plane and KVM nodes:

   1. Identify the KVM nodes:

      salt "kvm*" test.ping
      

      Example of system response:

      kvm01.bud.mirantis.net:
          True
      kvm03.bud.mirantis.net:
          True
      kvm02.bud.mirantis.net:
          True
      

      In the example above, three KVM nodes are identified: kvm01, kvm02, and kvm03.

   2. Identify the VMs running on top of each KVM node. For example, for kvm01:

      salt "kvm01*" cmd.run "virsh list --name"
      

      Example of system response:

      kvm01.bud.mirantis.net:
          cfg01
          prx01.<domain_name>
          ntw01.<domain_name>
          msg01.<domain_name>
          dbs01.<domain_name>
          ctl01.<domain_name>
          cid01.<domain_name>
          nal01.<domain_name>
      

   3. Using the output of the previous command, upgrade the cid, ctl, gtw/ntw/nal, log, mon, msg, mtr, dbs, and prx VMs of the particular KVM node. Do not upgrade cfg01, cmp, and kvm.

      Note

      Ceph nodes are out of the scope of this procedure.

      Example:

      for NODE in prx01.<domain_name> \
                  ntw01.<domain_name> \
                  msg01.<domain_name> \
                  dbs01.<domain_name> \
                  ctl01.<domain_name> \
                  cid01.<domain_name> \
                  nal01.<domain_name>
      do
        salt "${NODE}*" cmd.run "export DEBIAN_FRONTEND=noninteractive && \
                        apt-get update && \
                        apt-get -y upgrade && \
                        apt-get -y -o Dpkg::Options::="--force-confdef" \
                                -o Dpkg::Options::="--force-confnew" dist-upgrade"
      done
      

   4. Wait for all services of the cluster to be up and running.

      1. If the KVM node hosts GlusterFS, verify the GlusterFS server and volumes statuses as described in Troubleshoot GlusterFS. Proceed with further steps only if the GlusterFS status is healthy.
      2. If the KVM node hosts a dbs node, verify that the Galera cluster status is Synced and contains at least three nodes as described in Verify a Galera cluster status.
      3. If the KVM node hosts a msg node, verify the RabbitMQ cluster status and that it contains at least three nodes as described in Troubleshoot RabbitMQ.
      4. If the KVM node hosts OpenContrail 4.x, verify that its services are up and running as described in Verify the OpenContrail status. If any service fails, troubleshoot it as required. For details, see: Troubleshoot OpenContrail.

   5. Once you upgrade the VMs running on top of a KVM node, upgrade and restart this KVM node itself. For example, kvm01:

      salt "kvm01*" cmd.run "export DEBIAN_FRONTEND=noninteractive && \
                        apt-get update && \
                        apt-get -y upgrade && \
                        apt-get -y -o Dpkg::Options::="--force-confdef" \
                                   -o Dpkg::Options::="--force-confnew" dist-upgrade && \
                        shutdown -r 0"
      

   6. Wait for all services of the cluster to be up and running. For details, see substeps of the step 3.4.

   7. Repeat the steps 3.1-3.7 for the remaining kvm nodes one by one.

4. Upgrade the OpenStack compute nodes using the example below, where cmp10. is a set of nodes.

   Caution

   Before upgrading a particular set of cmp nodes, migrate the critical cloud environment workloads, which should not be powered off, from these nodes to the cmp nodes that are not under maintenance.

   Example:

   # Check that the command will be applied to only needed nodes
   salt -E "cmp10." test.ping
   # Perform the upgrade
   salt -E "cmp10." cmd.run "export DEBIAN_FRONTEND=noninteractive && apt-get \
   update && apt-get -y upgrade && apt-get -y -o \
   Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" \
   dist-upgrade && apt-get -y install linux-headers-generic && shutdown -r 0"