Upgrade RabbitMQ¶

This section instructs you on how to upgrade the RabbitMQ component automatically through Jenkins using the Deploy - upgrade RabbitMQ server pipeline job.

Prerequisites to upgrade RabbitMQ to version 3.8.2:

1. Log in to the Salt Master node.

2. Verify that the salt-formula-rabbitmq version is 0.2+202006030849.6079bf0~xenial1 or newer:

   dpkg -l |grep salt-formula-rabbitmq
   

3. Verify that the salt-formula-oslo-templates version is 2018.1+202006191406.b3839c0~xenial1 or newer.

   dpkg -l |grep salt-formula-oslo-templates
   

4. Perform the following steps depending on your OpenStack version:

   • For OpenStack Queens, verify that the following OpenStack configuration variables are not lower than specified:

     rabbit_retry_interval = 5
     rabbit_retry_backoff = 10
     kombu_reconnect_delay = 5.0
     

     For example, to verify the value of rabbit_retry_backoff across all nodes:

     salt '*' cmd.run 'grep -r rabbit_retry_backoff /etc'
     

   • For OpenStack Pike, verify that the python-oslo.messaging package version is 5.30.8-1~u16.04+mcp19 or newer:

     salt '*' pkg.version 'python-oslo.messaging'
     

5. Verify that the RabbitMQ version 3.8.2 is available on the msg nodes:

   salt -C 'I@rabbitmq:server' cmd.run 'apt-cache policy rabbitmq-server'
   

To upgrade RabbitMQ:

1. Prepare the Neutron server for the RabbitMQ upgrade:

   Caution

   This step is required since the Neutron service is sensitive to the RabbitMQ stability. The following Neutron configuration prevents massive resource rescheduling that can lead to the unbalanced load on the gateway nodes and other undesired consequences.

   1. On each OpenStack controller node, modify the neutron.conf file as follows:

      allow_automatic_dhcp_failover = false
      allow_automatic_l3agent_failover = false
      

   2. Restart the neutron-server service:

      service neutron-server restart
      

2. For the large clusters with more than 50 nodes and more than 100 Open vSwitch ports per node, stop the Neutron Open vSwitch agents on each gateway and compute node. This prevents overloading of the Neutron servers with massive agent resyncs.

   service neutron-openvswitch-agent stop
   

3. Log in to the Salt Master node.

4. Open the cluster level of your deployment model.

5. Include the RabbitMQ upgrade pipeline job to DriveTrain:

   1. Add the following class to classes/cluster/<cluster_name>/cicd/control/leader.yml:

      classes:
      - system.jenkins.client.job.deploy.update.upgrade_rabbitmq
      

   2. Apply the jenkins.client state on the Jenkins nodes:

      salt -C 'I@jenkins:client' state.sls jenkins.client
      

   3. Set the parameters in classes/cluster/<cluster_name>/infra/init.yml as follows:

      parameters:
        _param:
          openstack_upgrade_enabled: true
      

   4. Refresh pillars on the msg* nodes:

      salt 'msg*' saltutil.refresh_pillar
      

6. Apply the linux.system.repo state on the msg* nodes to add repositories with new RabbitMQ packages:

   salt 'msg*' state.sls linux.system.repo
   

7. Log in to the Jenkins web UI.

8. Open the Deploy - upgrade RabbitMQ server pipeline.

9. Specify the following parameters as required:

   Deploy - upgrade RabbitMQ server pipeline parameters¶

   Parameter	Description
   INTERACTIVE	Mode to get the detailed description of the pipeline job flow through the stages
   OS_DIST_UPGRADE	Upgrades system packages including kernel, aka apt-get dist-upgrade. Optional, launches only if OS_DIST_UPGRADE is selected.
   OS_UPGRADE	Upgrades all installed applications, aka apt-get upgrade. Optional, launches only if OS_UPGRADE is selected.
   SALT_MASTER_CREDENTIALS	Defines the Salt Master node credentials to use for connection, defaults to salt.
   SALT_MASTER_URL	Defines the Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.
   TARGET_SERVERS	Adds the target RabbitMQ nodes. Defaults to msg*.

10. Click Deploy.

    To monitor the deployment process, follow the instructions in MCP Deployment Guide: View the deployment details.

    The pipeline Deploy - upgrade RabbitMQ server workflow¶

    Stage	Description
    Pre-upgrade	Only non-destructive actions are applied during this phase. Basic service verification is performed. The job is launched on all target servers before moving to the next stage.
    Stop RabbitMQ service	All RabbitMQ services are stopped on the TARGET_SERVERS nodes in the reverse order. For instance, msg03, msg02, and then msg01. OpenStack APIs are not accessible starting from this point. This does not affect the data plane services such as OVS or KVM.
    Upgrade RabbitMQ	RabbitMQ and Erlang code are upgraded. No workload downtime is expected.
    Upgrade OS	Optional. Launches only if OS_UPGRADE or OS_DIST_UPGRADE is selected. A reboot can be performed if required. When the node is back online, the basic service checks are performed.
    Start RabbitMQ service	All Rabbitmq services are running on the TARGET_SERVERS nodes starting with the last instance stooped. For example, msg01, msg02, and then msg03.

11. Starting from the 2019.2.10 update, apply the fluentd state to the msg nodes:

    salt -C 'I@rabbitmq:server' state.apply fluentd
    

12. Revert the changes in the neutron.conf file made during step 1.

13. Gradually start the Neutron agents if you have stopped them during step 2:

    service neutron-dhcp-agent start
    service neutron-l3-agent start
    service neutron-metadata-agent start
    service neutron-openvswitch-agent start
    

14. Revert the changes in the classes/cluster/<cluster_name>/infra/init.yml file made during step 5.3.