Alertmanager

Alertmanager

This section describes the alerts for the Alertmanager service.


AlertmanagerNotificationFailureWarning

Severity

Warning

Summary

Alertmanager has {{ $value }} failed notifications for {{$labels.integration }} on the {{ $labels.instance}} instance for 2 minutes.

Raise condition

increase(alertmanager_notifications_failed_total[2m]) > 0

Description

Raises when Alertmanager fails to send a notification to a specific receiver or channel for the following exemplary reasons:

  • Slack: wrong API key or channel name

  • Email: authentication issues or the SMTP server is not available

  • Webhook: the server is not available

Troubleshooting

Run docker service logs monitoring_alertmanager on any mon node and inspect the Alertmanager logs.

Tuning

Not required

AlertmanagerAlertsInvalidWarning

Severity

Warning

Summary

An average of {{ $value }} Alertmanager {{$labels.integration }} alerts on the {{ $labels.instance }} instance are invalid for 2 minutes.

Raise condition

increase(alertmanager_alerts_invalid_total[2m]) > 0

Description

Raises when Alertmanager receives an alert with errors, for example:

  • Missing start and end time

  • Empty labels

  • Wrong labels or annotations. The key or value must not be empty, must start with a char and must be alphanumeric.

Troubleshooting

Run docker service logs monitoring_alertmanager on any mon node and inspect the Alertmanager logs.

Tuning

Not required