Perform the post-upgrade activities¶

The post-upgrade activities include the post-upgrade testing cycle and cleanup.

To finalize the upgrade:

1. Perform the full verification cycle of your MCP OpenStack deployment.

2. Verify that the following variables are set in the classes/cluster/<cluster_name>/infra/init.yml file:

   parameters:
     _param:
       openstack_upgrade_enabled: false
       openstack_version: queens
       openstack_old_version: pike
   

3. (Optional) If Gnocchi was upgraded, define the following parameters in classes/cluster/<cluster_name>/openstack/init.yml:

   parameters:
     _param:
       gnocchi_version: 4.2
       gnocchi_old_version: 4.0
   

4. Refresh pillars:

   salt '*' saltutil.refresh_pillar
   

5. Remove the test workloads/monitoring.

6. Remove the upgrade leftovers that were created by applying the <app>.upgrade.post state:

   1. Log in to the Salt Master node.

   2. Get the list of all upgradable OpenStack components. For example:

      salt cfg01* config.get orchestration:upgrade:applications --out=json
      

      Example of system response:

      {
          "<model_of_salt_master_name>": {
              "nova": {
                  "priority": 1100
              },
              "heat": {
                  "priority": 1250
              },
              "keystone": {
                  "priority": 1000
              },
              "horizon": {
                  "priority": 1800
              },
              "cinder": {
                  "priority": 1200
              },
              "glance": {
                  "priority": 1050
              },
              "neutron": {
                  "priority": 1150
              },
              "designate": {
                  "priority": 1300
              }
          }
      }
      

   3. Range the components from the output by priority. For example:

      keystone
      glance
      nova
      neutron
      cinder
      heat
      designate
      horizon
      

   4. Get the list of all target nodes:

      salt-key | grep $cluster_domain | \
      grep -v $salt_master_hostname | tr '\n' ' '
      

      Note

      The cluster_domain variable stands for the name of the domain used as part of the cluster FQDN. For details, see MCP Deployment guide: General deployment parameters: Basic deployment parameters

      The salt_master_hostname variable stands for the hostname of the Salt Master node and is cfg01 by default. For details, see MCP Deployment guide: Infrastructure related parameters: Salt Master

   5. For each target node, get the list of installed applications:

      salt <node_name> pillar.items __reclass__:applications --out=json
      

   6. Match the lists of upgradable OpenStack components with the lists of installed applications for each target node.

   7. Apply the following states to each target node for each installed application in strict order of priority:

      salt <node_name> state.apply <component_name>.upgrade.post
      

      For example, for Nova installed on the cmp01 compute node, run:

      salt cmp01 state.apply nova.upgrade.post
      

   Note

   On the clouds of medium and large sizes, you may want to automate this step. Use the following script as an example of possible automatization. Before running the script, verify that you define the $cluster_domain and $salt_master_hostname variables.

    #!/bin/bash
   #List of formulas that implements upgrade API sorted by priority
   all_formulas=$(salt cfg01* config.get orchestration:upgrade:applications   --out=json | \
   jq '.[] | . as $in | keys_unsorted | map ({"key": ., "priority": $in[.].priority}) | sort_by(.priority) | map(.key | [(.)]) | add' | \
   sed -e 's/"//g' -e 's/,//g' -e 's/\[//g' -e 's/\]//g')
   #List of nodes in cloud
   list_nodes=`salt-key | grep $cluster_domain | grep -v $salt_master_hostname | tr '\n' ' '`
   for node in $list_nodes; do
     #List of applications on the given node
     node_applications=$(salt $node pillar.items __reclass__:applications   --out=json | \
     jq 'values |.[] | values |.[] | .[]' | tr -d '"' | tr '\n' ' ')
     for component in $all_formulas ; do
       if [[ " ${node_applications[*]} " == *"$component"* ]]; then
         salt $node state.apply $component.upgrade.post
       fi
     done
   done
   

7. Set the following variables in classes/cluster/<cluster_name>/infra/init.yml:

   parameters:
     _param:
       openstack_upgrade_enabled: false
       openstack_version: queens
       openstack_old_version: queens
   

   If Gnocchi was upgraded, set the following parameters in the same file:

   parameters:
     _param:
       gnocchi_version: 4.2
       gnocchi_old_version: 4.2
   

8. Refresh pillars:

   salt '*' saltutil.refresh_pillar