Troubleshoot RabbitMQ

Troubleshoot RabbitMQ

To troubleshoot RabbitMQ:

  1. Log in to any OpenStack messaging node.

  2. Verify the RabbitMQ cluster status:

    rabbitmqctl cluster_status
    
  3. Verify the AMQP messaging RabbitMQ status:

    curl -s -o rabbitmqadmin http://$(hostname -s):15672/cli/rabbitmqadmin;
    chmod 0755 ./rabbitmqadmin
    adm_creds=$(salt-call pillar.get rabbitmq:server:admin --out=yaml \
      | egrep ':\ (.+)' | tr '\n' ' ' | sed 's/name: /-u /; s/password: /-p /')
    ./rabbitmqadmin -H $(hostname -s) $adm_creds show overview
    ./rabbitmqadmin -H $(hostname -s) $adm_creds list queues
    ./rabbitmqadmin -H $(hostname -s) $adm_creds list queues vhost name node messages \
      message_stats.publish_details.rate | grep -v " 0"
    ./rabbitmqadmin -H $(hostname -s) $adm_creds -V /openstack list queues vhost name \
    node messages message_stats.publish_details.rate | grep -v " 0"
    
  4. List the number of messages and consumers for each queue. Select vhost, otherwise, you will work with the default queue that is /. The number of messages should be low and number of consumers above 0:

    rabbitmqctl list_queues -p /openstack messages consumers name
    
  5. Print stuck nodes if any:

    rabbitmqctl eval 'rabbit_diagnostics:maybe_stuck().'