Upgrade the OpenContrail nodes

Upgrade the OpenContrail nodes

After you prepare the cluster model of your MCP cluster, proceed to upgrading the OpenContrail controller nodes and the OpenContrail vRouter packages on the compute nodes.

Warning

During the upgrade process, the following resources are affected:

  • The instance(s) running on the compute nodes can be unavailable for up to 30 seconds.

  • The creation of new instances is not possible during the same time interval.

Therefore, you must plan a maintenance window as well as test the upgrade before applying it to production.

To upgrade the OpenContrail nodes:

  1. Log in to the Jenkins web UI.

  2. Open the Deploy - upgrade Opencontrail to 4.x pipeline.

  3. Specify the following parameters:

    Parameter

    Description and values

    SALT_MASTER_CREDENTIALS

    The Salt Master credentials to use for connection, defaults to salt.

    SALT_MASTER_URL

    The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.

    STAGE_CONTROLLERS_UPGRADE

    Select this check box to run upgrade on the OpenContrail controller nodes.

  4. Click Deploy.

  5. If the pipeline fails with a Docker container with contrail-control service not starting or infinite restarting:

    1. Log in to the container of the affected OpenContrail node.

    2. Verify whether the logs contain errors UID already exists or GID already exists. For example:

      docker logs --tail 10 --follow --timestamps <container_ID>
      

      Example of system response extract:

      2018-09-13T06:52:15.569311246Z usermod: UID '109' already exists
      2018-09-13T06:53:15.714037288Z usermod: UID '109' already exists
      2018-09-13T06:54:15.863864322Z usermod: UID '109' already exists
      2018-09-13T06:55:16.026689446Z usermod: UID '109' already exists
      
    3. If the logs contain the above-mentioned errors:

      1. Log in to one of the affected OpenContrail nodes.

      2. Replace the contrail GID and UID with new ones that are not used by existing group and user. For example, replace 109 with 901. Run the following commands:

        usermod -u <new uid> contrail
        groupmod -g <new gid> contrail
        
      3. Rerun the Deploy - upgrade Opencontrail to 4.x pipeline.

  6. If the pipeline fails with some OpenContrail services stuck in the initializing state:

    1. Restart ZooKeeper on all analyticsdb containers.

    2. Once ZooKeeper is up and running, restart the services that are stuck.

    3. Rerun the Deploy - upgrade Opencontrail to 4.x pipeline.

  7. If the pipeline fails with the Command ‘docker-compose up -d’ failed error message that causes docker-compose fail to initialize the opencontrail_analytics_1 and opencontrail_analyticsdb_1 containers:

    1. Run the following command from the Salt Master node:

      salt -C 'nal*' cmd.run "rm -rf /etc/cassandra/cassandra_analytics.yaml \
      && rm -rf /etc/cassandra/cassandra-env-analytics.sh/ \
      && rm -rf /etc/zookeeper/conf/zoo_analytics.cfg"
      
    2. Rerun the Deploy - upgrade Opencontrail to 4.x pipeline.

To upgrade the OpenContrail vRouter packages on the compute nodes:

  1. Log in to the Jenkins web UI.

  2. Open the Deploy - upgrade Opencontrail to 4.x pipeline.

  3. Specify the following parameters:

    Parameter

    Description and values

    COMPUTE_TARGET_SERVERS

    Add I@opencontrail:compute or target the global name of your compute nodes, for example cmp001*.

    COMPUTE_TARGET_SUBSET_LIVE

    Add 1 to run the upgrade first on only one of the nodes defined in the COMPUTE_TARGET_SERVERS field. After this stage is done, in the upgrade pipeline you will be asked to continue the upgrade all nodes defined in the COMPUTE_TARGET_SERVERS field.

    SALT_MASTER_CREDENTIALS

    The Salt Master credentials to use for connection, defaults to salt.

    SALT_MASTER_URL

    The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.

    STAGE_COMPUTES_UPGRADE

    Select this check box to run upgrade on the compute nodes.

  4. Click Deploy. For details how to monitor the deployment process, see: MCP Deployment Guide: View the deployment details.

The Deploy - upgrade Opencontrail to 4.x pipeline upgrade stages are as follows:

  1. If STAGE_COMPUTES_UPGRADE is selected, the pipeline upgrades the OpenContrail packages on the compute nodes in two iterations:

    1. Upgrade the sample nodes defined by COMPUTE_TARGET_SUBSET_LIVE.

    2. After a manual confirmation, upgrade all compute nodes targeted by COMPUTE_TARGET_SERVERS.

  2. If STAGE_CONTROLLERS_UPGRADE is selected, the pipeline stops the OpenContrail services on the OpenContrail nal nodes and starts the analytics and analytics db containers. Once done, the same procedure applies to the OpenContrail ntw nodes.