Ceph¶

The Ceph cluster is in the WARNING state. For details, run ceph -s.

Raises according to the status reported by the Ceph cluster.

Not required

The Ceph cluster is in the CRITICAL state. For details, run ceph -s.

Raises according to the status reported by the Ceph cluster.

Not required

{{ $value }}% of Ceph Monitors are down. For details, run ceph -s.

Raises if any of the Ceph Monitors in the Ceph cluster is down.

Not required

{{ $value }}% of Ceph OSDs are down. For details, run ceph osd tree.

Raises if any of the Ceph OSD nodes in the Ceph cluster is down.

Not required

{{ $value }} bytes of the Ceph OSD space (>=75%) is used for 3 minutes. For details, run cephdf.

Raises when a Ceph OSD used space capacity exceeds the threshold of 75%.

For example, to change the threshold to 80%:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           CephOsdSpaceUsageWarning:
             if: >-
               ceph_cluster_total_used_bytes > ceph_cluster_total_bytes * 0.8
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

{{ $value }} bytes of the Ceph OSD space (>=85%) is used for 3 minutes. For details, run cephdf.

Raises when a Ceph OSD used space capacity exceeds the threshold of 85%.

For example, to change the threshold to 95%:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           CephOsdSpaceUsageMajor:
             if: >-
               ceph_cluster_total_used_bytes > ceph_cluster_total_bytes * 0.95
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

The Ceph {{pool_name}} pool uses 75% of available space for 3 minutes. For details, run ceph df.

Raises when a Ceph pool used space capacity exceeds the threshold of 75%.

Should be tuned per pool. For example, to change the threshold to 80% for pool volumes:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           CephPoolvolumesSpaceUsageWarning:
             if: >-
               ceph_pool_bytes_used / (ceph_pool_bytes_used + ceph_pool_max_avail) * \
               on(pool_id) group_left(name) ceph_pool_metadata{name="volumes"} > 0.8
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

The Ceph {{pool_name}} pool uses 85% of available space for 3 minutes. For details, run ceph df.

Raises when a Ceph pool used space capacity exceeds the threshold of 85%.

Should be tuned per pool. For example, to change the threshold to 90% for pool volumes:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           CephPoolvolumesSpaceUsageCritical:
             if: >-
               ceph_pool_bytes_used / (ceph_pool_bytes_used + ceph_pool_max_avail) * \
               on(pool_id) group_left(name) ceph_pool_metadata{name="volumes"} > 0.9
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

Some Ceph OSDs contain more than 200 PGs. This may have a negative impact on the cluster performance. For details, run ceph pg dump.

Raises when the number of PGs on Ceph OSDs is higher than the default threshold of 200.

For example, to change the threshold to 400 PGs:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           CephOsdPgNumTooHighWarning:
             if: >-
               max(ceph_osd_numpg) > 400
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

Some Ceph OSDs contain more than 300 PGs. This may have a negative impact on the cluster performance. For details, run ceph pg dump.

Raises when the number of PGs on Ceph OSDs is bigger than the default threshold of 300.

For example, to change the threshold to 500 PGs:

1. On the cluster level of the Reclass model, create a common file for all alert customizations. Skip this step to use an existing defined file.

   1. Create a file for alert customizations:

      touch cluster/<cluster_name>/stacklight/custom/alerts.yml
      

   2. Define the new file in cluster/<cluster_name>/stacklight/server.yml:

      classes:
      - cluster.<cluster_name>.stacklight.custom.alerts
      ...
      

2. In the defined alert customizations file, modify the alert threshold by overriding the if parameter:

   parameters:
     prometheus:
       server:
         alert:
           CephOsdPgNumTooHighCritical:
             if: >-
               max(ceph_osd_numpg) > 500
   

3. From the Salt Master node, apply the changes:

   salt 'I@prometheus:server' state.sls prometheus.server
   

4. Verify the updated alert definition in the Prometheus web UI.

The IOPS on the {{ $labels.ceph_daemon }} Ceph OSD are increasing rapidly.

Predicts the IOPS consumption per Ceph OSD in a specified time range, 1 week by default. The threshold parameter defines the time range.

The IOPS on the {{ $labels.ceph_daemon }} Ceph OSD are increasing rapidly.

Predicts the IOPS consumption per OSD in a specified time range, 1 week by default. The threshold parameter defines the time range.

The {{$labels.host}} host may run out of available RAM next week.

Predicts the exhaustion of the available RAM on Ceph nodes in a defined time range.

The {{$labels.name}} on the {{$labels.host}} host may become unresponsive shortly. Verify the OSDs top load on the Ceph OSD Overview Grafana dashboard.

Predicts the OSD disks responsiveness in a specified time range based on the write latency. The threshold parameter defines the time range. The write_latency_threshold parameter defines the differences to detect in the write latency.

The {{$labels.name}} on the {{$labels.host}} host may become unresponsive shortly. Verify the OSDs top load on the Ceph OSD Overview Grafana dashboard.

Predicts the OSD disks responsiveness in a specified time range based on the read latency. The threshold parameter defines the time range. The read_latency_threshold parameter defines the differences to detect in the read latency.

The {{pool_name}} pool may consume more than {{100*space_threshold}}% of the available capacity in 1 week. For details, run ceph df and plan proper actions.

Predicts the exhaustion of all available capacity of a pool in a defined time range. The threshold parameter specifies the time range to use. The space_threshold parameter defines the capacity threshold, similar to the one set in CephPool{pool_name}SpaceUsageCritical.

The IOPS in the {{pool_name}} are increasing rapidly.

Predicts the IOPS consumption per pool in a specified time range, 1 week by default. The threshold parameter specifies the time range to use.

The IOPS in the {{pool_name}} are increasing rapidly.

Predicts the IOPS utilisation per pool in a specified time range, 1 week by default. The threshold parameter specifies the time range to use.

RADOS Gateway outage.

Raises if RADOS Gateway is not accessible for all available RADOS Gateway endpoints in the OpenStack service catalog.

The {{ $labels.name }} endpoint is not accessible.

Raises if RADOS Gateway is not accessible for the {{ $labels.name }} endpoint.