Prometheus¶

The Prometheus target for the {{ $labels.job }} job on the {{ $labels.host or $labels.instance }} node is down for 2 minutes.

Raises when Prometheus fails to scrape a target for 2 minutes. The reasons depend on the target type. For example, Telegraf-related or connectivity issues, Fluentd misconfiguration, issues with libvirt or JMX Exporter.

Not required

{{ $value }} Prometheus samples on the {{$labels.instance }} instance are out of order (as measured over the last minute).

Raises when Prometheus observes time series samples with a wrong order.

{{ $value }} Prometheus samples on the {{$labels.instance }} instance have time stamps out of bounds (as measured over the last minute).

Raises when Prometheus observes samples with time stamps greater than the current time.

{{ $value }} Prometheus samples on the {{$labels.instance }} instance have duplicate time stamps (as measured over the last minute).

Raises when Prometheus observes samples with duplicated time stamps.

The Prometheus service writes on the {{$labels.instance }} instance do not keep up with the data ingestion speed for 10 minutes.

Raises when the Prometheus service writes do not keep up with the data ingestion speed for 10 minutes.

The Prometheus remote storage queue on the {{$labels.instance }} instance is 75% full for 2 minutes.

Raises when Prometheus remote write queue is 75% full.

For example, to change the warning threshold to 90%:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           PrometheusRemoteStorageQueueFullWarning:
             if: >-
               prometheus_remote_storage_queue_length / \
               prometheus_remote_storage_queue_capacity * 100 > 90
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

The Prometheus Relay service on the {{$labels.host}} node is down for 2 minutes.

Raises when Telegraf cannot find running prometheus-relay processes on any mtr host.

Not required

More than 50% of Prometheus Relay services are down for 2 minutes.

Raises when Telegraf cannot find running prometheus-relay processes on more than 50% of the mtr hosts.

Not required

All Prometheus Relay services are down for 2 minutes.

Raises when Telegraf cannot find running prometheus-relay processes on all mtr hosts.

Not required

The Prometheus long-term storage service on the {{$labels.host}} node is down for 2 minutes.

Raises when Telegraf cannot find running prometheus processes on any mtr host.

Not required

More than 50% of the Prometheus long-term storage services are down for 2 minutes.

Raises when Telegraf cannot find running prometheus processes on more than 50% of the mtr hosts.

Not required

All Prometheus long-term storage services are down for 2 minutes.

Raises when Telegraf cannot find running prometheus processes on all mtr hosts.

Not required

The Prometheus server for the {{ $labels.job }} job on the {{ or $labels.host $labels.instance }} node has failed evaluations for recording rules. Verify the rules state in the Status/Rules section of the Prometheus web UI.

Raises when the number of evaluation failures of Promethues recording rules continuously increases for 10 minutes. The issue typically occurs once you reload the Prometheus service after configuration changes.

Not required