Upgrade the OpenContrail nodes¶

After you prepare the cluster model of your MCP cluster, proceed to upgrading the OpenContrail controller nodes and the OpenContrail vRouter packages on the compute nodes.

To upgrade the OpenContrail nodes:

1. Log in to the Jenkins web UI.

2. Open the Deploy - upgrade Opencontrail to 4.x pipeline.

3. Specify the following parameters:

   Parameter	Description and values
   SALT_MASTER_CREDENTIALS	The Salt Master credentials to use for connection, defaults to salt.
   SALT_MASTER_URL	The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.
   STAGE_CONTROLLERS_UPGRADE	Select this check box to run upgrade on the OpenContrail controller nodes.

4. Click Deploy.

5. If the pipeline fails with a Docker container with contrail-control service not starting or infinite restarting:

   1. Log in to the container of the affected OpenContrail node.

   2. Verify whether the logs contain errors UID already exists or GID already exists. For example:

      docker logs --tail 10 --follow --timestamps <container_ID>
      

      Example of system response extract:

      2018-09-13T06:52:15.569311246Z usermod: UID '109' already exists
      2018-09-13T06:53:15.714037288Z usermod: UID '109' already exists
      2018-09-13T06:54:15.863864322Z usermod: UID '109' already exists
      2018-09-13T06:55:16.026689446Z usermod: UID '109' already exists
      

   3. If the logs contain the above-mentioned errors:

      1. Log in to one of the affected OpenContrail nodes.

      2. Replace the contrail GID and UID with new ones that are not used by existing group and user. For example, replace 109 with 901. Run the following commands:

         usermod -u <new uid> contrail
         groupmod -g <new gid> contrail
         

      3. Rerun the Deploy - upgrade Opencontrail to 4.x pipeline.

6. If the pipeline fails with some OpenContrail services stuck in the initializing state:

   1. Restart ZooKeeper on all analyticsdb containers.
   2. Once ZooKeeper is up and running, restart the services that are stuck.
   3. Rerun the Deploy - upgrade Opencontrail to 4.x pipeline.

7. If the pipeline fails with the Command ‘docker-compose up -d’ failed error message that causes docker-compose fail to initialize the opencontrail_analytics_1 and opencontrail_analyticsdb_1 containers:

   1. Run the following command from the Salt Master node:

      salt -C 'nal*' cmd.run "rm -rf /etc/cassandra/cassandra_analytics.yaml \
      && rm -rf /etc/cassandra/cassandra-env-analytics.sh/ \
      && rm -rf /etc/zookeeper/conf/zoo_analytics.cfg"
      

   2. Rerun the Deploy - upgrade Opencontrail to 4.x pipeline.

To upgrade the OpenContrail vRouter packages on the compute nodes:

1. Log in to the Jenkins web UI.

2. Open the Deploy - upgrade Opencontrail to 4.x pipeline.

3. Specify the following parameters:

   Parameter	Description and values
   COMPUTE_TARGET_SERVERS	Add I@opencontrail:compute or target the global name of your compute nodes, for example cmp001*.
   COMPUTE_TARGET_SUBSET_LIVE	Add 1 to run the upgrade first on only one of the nodes defined in the COMPUTE_TARGET_SERVERS field. After this stage is done, in the upgrade pipeline you will be asked to continue the upgrade all nodes defined in the COMPUTE_TARGET_SERVERS field.
   SALT_MASTER_CREDENTIALS	The Salt Master credentials to use for connection, defaults to salt.
   SALT_MASTER_URL	The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.
   STAGE_COMPUTES_UPGRADE	Select this check box to run upgrade on the compute nodes.

4. Click Deploy. For details how to monitor the deployment process, see: MCP Deployment Guide: View the deployment details.

The Deploy - upgrade Opencontrail to 4.x pipeline upgrade stages are as follows:

1. If STAGE_COMPUTES_UPGRADE is selected, the pipeline upgrades the OpenContrail packages on the compute nodes in two iterations:
   1. Upgrade the sample nodes defined by COMPUTE_TARGET_SUBSET_LIVE.
   2. After a manual confirmation, upgrade all compute nodes targeted by COMPUTE_TARGET_SERVERS.
2. If STAGE_CONTROLLERS_UPGRADE is selected, the pipeline stops the OpenContrail services on the OpenContrail nal nodes and starts the analytics and analytics db containers. Once done, the same procedure applies to the OpenContrail ntw nodes.