Upgrade RabbitMQ

Upgrade RabbitMQ

This section instructs you on how to upgrade the RabbitMQ component automatically through Jenkins using the Deploy - upgrade RabbitMQ server pipeline job.

Note

  • This feature is available starting from the MCP 2019.2.4 maintenance update. Before enabling the feature, follow the steps described in Apply maintenance updates.

  • Starting from the MCP 2019.2.10 maintenance update, RabbitMQ version 3.8.2 is available. Prior to updating RabbitMQ to the new version, perform the steps described in Apply maintenance updates and the following prerequisite steps.

  • For OpenStack Pike environments with RabbitMQ version 3.8.2, the timeouts are set to rabbit_retry_interval = 5, rabbit_retry_backoff = 10, and kombu_reconnect_delay = 5.0 by default in the oslo.messaging package.

Caution

We recommend that you do not upgrade or update OpenStack and RabbitMQ simultaneously. Upgrade or update the RabbitMQ component only once OpenStack is running on the new version.

Prerequisites to upgrade RabbitMQ to version 3.8.2:

  1. Log in to the Salt Master node.

  2. Verify that the salt-formula-rabbitmq version is 0.2+202006030849.6079bf0~xenial1 or newer:

    dpkg -l |grep salt-formula-rabbitmq
    
  3. Verify that the salt-formula-oslo-templates version is 2018.1+202006191406.b3839c0~xenial1 or newer.

    dpkg -l |grep salt-formula-oslo-templates
    
  4. Perform the following steps depending on your OpenStack version:

    • For OpenStack Queens, verify that the following OpenStack configuration variables are not lower than specified:

      rabbit_retry_interval = 5
      rabbit_retry_backoff = 10
      kombu_reconnect_delay = 5.0
      

      For example, to verify the value of rabbit_retry_backoff across all nodes:

      salt '*' cmd.run 'grep -r rabbit_retry_backoff /etc'
      
    • For OpenStack Pike, verify that the python-oslo.messaging package version is 5.30.8-1~u16.04+mcp19 or newer:

      salt '*' pkg.version 'python-oslo.messaging'
      
  5. Verify that the RabbitMQ version 3.8.2 is available on the msg nodes:

    salt -C 'I@rabbitmq:server' cmd.run 'apt-cache policy rabbitmq-server'
    

To upgrade RabbitMQ:

  1. Prepare the Neutron server for the RabbitMQ upgrade:

    Caution

    This step is required since the Neutron service is sensitive to the RabbitMQ stability. The following Neutron configuration prevents massive resource rescheduling that can lead to the unbalanced load on the gateway nodes and other undesired consequences.

    1. On each OpenStack controller node, modify the neutron.conf file as follows:

      allow_automatic_dhcp_failover = false
      allow_automatic_l3agent_failover = false
      
    2. Restart the neutron-server service:

      service neutron-server restart
      
  2. For the large clusters with more than 50 nodes and more than 100 Open vSwitch ports per node, stop the Neutron Open vSwitch agents on each gateway and compute node. This prevents overloading of the Neutron servers with massive agent resyncs.

    service neutron-openvswitch-agent stop
    
  3. Log in to the Salt Master node.

  4. Open the cluster level of your deployment model.

  5. Include the RabbitMQ upgrade pipeline job to DriveTrain:

    1. Add the following class to classes/cluster/<cluster_name>/cicd/control/leader.yml:

      classes:
      - system.jenkins.client.job.deploy.update.upgrade_rabbitmq
      
    2. Apply the jenkins.client state on the Jenkins nodes:

      salt -C 'I@jenkins:client' state.sls jenkins.client
      
    3. Set the parameters in classes/cluster/<cluster_name>/infra/init.yml as follows:

      parameters:
        _param:
          openstack_upgrade_enabled: true
      
    4. Refresh pillars on the msg* nodes:

      salt 'msg*' saltutil.refresh_pillar
      
  6. Apply the linux.system.repo state on the msg* nodes to add repositories with new RabbitMQ packages:

    salt 'msg*' state.sls linux.system.repo
    
  7. Log in to the Jenkins web UI.

  8. Open the Deploy - upgrade RabbitMQ server pipeline.

  9. Specify the following parameters as required:

    Deploy - upgrade RabbitMQ server pipeline parameters

    Parameter

    Description

    INTERACTIVE

    Mode to get the detailed description of the pipeline job flow through the stages

    OS_DIST_UPGRADE

    Upgrades system packages including kernel, aka apt-get dist-upgrade. Optional, launches only if OS_DIST_UPGRADE is selected.

    OS_UPGRADE

    Upgrades all installed applications, aka apt-get upgrade. Optional, launches only if OS_UPGRADE is selected.

    SALT_MASTER_CREDENTIALS

    Defines the Salt Master node credentials to use for connection, defaults to salt.

    SALT_MASTER_URL

    Defines the Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.

    TARGET_SERVERS

    Adds the target RabbitMQ nodes. Defaults to msg*.

  10. Click Deploy.

    To monitor the deployment process, follow the instructions in MCP Deployment Guide: View the deployment details.

    The pipeline Deploy - upgrade RabbitMQ server workflow

    Stage

    Description

    Pre-upgrade

    Only non-destructive actions are applied during this phase. Basic service verification is performed. The job is launched on all target servers before moving to the next stage.

    Stop RabbitMQ service

    All RabbitMQ services are stopped on the TARGET_SERVERS nodes in the reverse order. For instance, msg03, msg02, and then msg01.

    OpenStack APIs are not accessible starting from this point. This does not affect the data plane services such as OVS or KVM.

    Upgrade RabbitMQ

    RabbitMQ and Erlang code are upgraded. No workload downtime is expected.

    Upgrade OS

    Optional. Launches only if OS_UPGRADE or OS_DIST_UPGRADE is selected. A reboot can be performed if required. When the node is back online, the basic service checks are performed.

    Start RabbitMQ service

    All Rabbitmq services are running on the TARGET_SERVERS nodes starting with the last instance stooped. For example, msg01, msg02, and then msg03.

  11. Starting from the 2019.2.10 update, apply the fluentd state to the msg nodes:

    salt -C 'I@rabbitmq:server' state.apply fluentd
    
  12. Revert the changes in the neutron.conf file made during step 1.

  13. Gradually start the Neutron agents if you have stopped them during step 2:

    service neutron-dhcp-agent start
    service neutron-l3-agent start
    service neutron-metadata-agent start
    service neutron-openvswitch-agent start
    
  14. Revert the changes in the classes/cluster/<cluster_name>/infra/init.yml file made during step 5.3.