Octavia¶

Octavia API is not accessible for all available Octavia endpoints in the OpenStack service catalog for 2 minutes.

Raises when the checks against one available internal Octavia endpoint in the OpenStack service catalog does not pass. Telegraf sends HTTP requests to the URLs from the OpenStack service catalog and compares the expected and actual HTTP response codes. The expected response code for Octavia is 200. For a list of all available endpoints, run openstack endpoint list.

Not required

The average per-second rate of errors in Octavia logs on the {{ $labels.host }} node is more than 0.2 error messages per second (as measured over the last 5 minutes).

Raises when the average per-second rate of the error, fatal, or emergency messages in the Octavia logs on the node is more than 0.2 per second. Fluentd forwards all logs from Octavia to Elasticsearch and counts the number of log messages per severity. The host label in the raised alert contains the host name of the affected node.

Typically, you should not change the default value. If the alert is constantly firing, inspect the Octavia error logs in the Kibana web UI. However, you can adjust the threshold to an acceptable error rate for a particular environment. In the Prometheus Web UI, use the raise condition query to view the appearance rate of a particular message type in logs for a longer period of time and define the best threshold.

For example, to change the threshold to 0.4:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           OctaviaErrorLogsTooHigh:
             if: >-
               sum(rate(log_messages{service=""octavia"", level=~""(?i:\
               (error|emergency|fatal))""}[5m])) without (level) > 0.4
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.