Silence alerts¶

Due to the Alertmanager issue, silences with regexp matchers do not mute all notifications for all alerts matched by the specified regular expression.

If you need to mute multiple alerts, for example, for maintenance or before cluster update, Mirantis recommends using a set of fixed-matcher silences instead. As an example, this section describes how to silence all alerts for a specified period through the Alertmanager web UI or CLI without using the regexp matchers. You can also manually force silence expiration before the specified period ends.

To silence all alerts:

Silence alerts through the Alertmanager web UI:
1. Log in to the Alertmanager web UI as described in Access StackLight web UIs.
2. Click New Silence.
3. Create four Prometheus Alertmanager silences. In Matchers, set Name to severity and Value to warning, minor, major, and critical, one for each silence.
  
  Note
  
  To silence the Watchdog alert, create an additional silence with severity set in Name and informational set in Value.

Silence alerts through CLI:

Run the following command setting the required duration:

kubectl exec -it -n stacklight prometheus-alertmanager-1 prometheus-alertmanager -- sh -c 'rm -f /tmp/all_silences; \
 touch /tmp/all_silences; \
 for severity in warning minor major critical; do \
   echo $severity; \
   amtool silence add severity=${severity} \
     --alertmanager.url=<http://prometheus-alertmanager> \
     --comment="silence them all" \
     --duration="2h" | tee /tmp/all_silences; \
 done'

Note

To silence the Watchdog alert, add informational to the list of severities.

To exprire alert silences:

To expire alert silences through the Alertmanager web UI, click Expire next to each silence.

To expire alert silences through CLI, run the following command:

kubectl exec -it -n stacklight prometheus-alertmanager-1 prometheus-alertmanager -- sh -c 'for silence in $(cat /tmp/all_silences); do \
    echo $severity; \
    amtool silence expire $silence \
      --alertmanager.url=<http://prometheus-alertmanager;> \
  done'