HAProxy
This section describes the alerts for the HAProxy service.
HaproxyServiceDown
| Severity |
Minor |
|---|
| Summary |
The HAProxy service on the {{ $labels.host }} node is down. |
|---|
| Raise condition |
haproxy_up != 1 |
|---|
| Description |
Raises when the HAProxy service on a node does not respond to Telegraf,
typically meaning that the HAproxy process is in the DOWN state on
that node. The host label in the raised alert contains the host name
of the affected node. |
|---|
| Troubleshooting |
- Verify the HAProxy status by running
systemctl status haproxy on
the affected node.
- If HAProxy is up and running, inspect the Telegraf logs on the
affected node using
journalctl -u telegraf.
|
|---|
| Tuning |
Not required |
|---|
HaproxyServiceDownMajor
| Severity |
Major |
|---|
| Summary |
More than 50% of HAProxy services within the {{ $labels.cluster }}
cluster are down. |
|---|
| Raise condition |
count(label_replace(haproxy_up, "cluster", "$1", "host",
"([^0-9]+).+") != 1) by (cluster) >= 0.5 *
count(label_replace(haproxy_up, "cluster", "$1", "host", "([^0-9]+).+"))
by (cluster) |
|---|
| Description |
Raises when the HAProxy service does not respond to Telegraf on more
than 50% of cluster nodes. The cluster label in the raised alert
contains the cluster prefix, for example, ctl, dbs, or mon. |
|---|
| Troubleshooting |
- Inspect the
HaproxyServiceDown alerts for the host names of the
affected nodes.
- Inspect
dmesg and /var/log/kern.log.
- Inspect the logs in
/var/log/haproxy.log.
- Inspect the Telegraf logs using
journalctl -u telegraf.
|
|---|
| Tuning |
Not required |
|---|
HaproxyServiceOutage
| Severity |
Critical |
|---|
| Summary |
All HAProxy services within the {{ $labels.cluster }} cluster are
down. |
|---|
| Raise condition |
count(label_replace(haproxy_up, "cluster", "$1", "host",
"([^0-9]+).+") != 1) by (cluster) == count(label_replace(haproxy_up,
"cluster", "$1", "host", "([^0-9]+).+")) by (cluster) |
|---|
| Description |
Raises when the HAProxy service does not respond to Telegraf on all
nodes of a cluster, typically indicating deployment or configuration
issues. The cluster label in the raised alert contains
the cluster prefix, for example, ctl, dbs, or mon. |
|---|
| Troubleshooting |
- Inspect the
HaproxyServiceDown alerts for the host names of the
affected nodes.
- Inspect
dmesg and /var/log/kern.log.
- Inspect the logs in
/var/log/haproxy.log.
- Inspect the Telegraf logs using
journalctl -u telegraf.
|
|---|
| Tuning |
Not required |
|---|
HaproxyHTTPResponse5xxTooHigh
| Severity |
Warning |
|---|
| Summary |
The average per-second rate of 5xx HTTP errors on the
{{ $labels.host }} node for the {{ $labels.proxy }} back end is
{{ $value }} (as measured over the last 2 minutes). |
|---|
| Raise condition |
rate(haproxy_http_response_5xx{sv="FRONTEND"}[2m]) > 1 |
|---|
| Description |
Raises when the HTTP 5xx responses sent by HAProxy increased for the
last 2 minutes, indicating a configuration issue with the HAProxy
service or back-end servers within the cluster. The host label in
the raised alert contains the host name of the affected node. |
|---|
| Troubleshooting |
Inspect the HAproxy logs by running journalctl -u haproxy on the
affected node and verify the state of the back-end servers. |
|---|
| Tuning |
Not required |
|---|
HaproxyBackendDown
| Severity |
Minor |
|---|
| Summary |
The {{ $labels.proxy }} back end on the {{ $labels.host }} node
is down. |
|---|
| Raise condition |
increase(haproxy_chkdown{sv="BACKEND"}[1m]) > 0 |
|---|
| Description |
Raises when an internal HAProxy check for the back-end availability
reported the back-end outage. The host and proxy labels in the
raised alert contain the host name of the affected node and the service
proxy name. |
|---|
| Troubleshooting |
- Inspect the HAProxy logs by running
journalctl -u haproxy on the
affected node.
- Verify the state of the affected back-end server:
- Verify that the server is responding and the back-end service is
active and responsive.
- Verify the state of the back-end service using an HTTP GET request,
for example,
curl -XGET http://ctl01:8888/. Typically, the
200 response code indicates the healthy state.
|
|---|
| Tuning |
Not required |
|---|
HaproxyBackendDownMajor
| Severity |
Major |
|---|
| Summary |
More than 50% of {{ $labels.proxy }} backends are down. |
|---|
| Raise condition |
- In 2019.2.10 and prior:
0.5 * avg(sum(haproxy_active_servers{type=""server""}) by (host,
proxy) + sum(haproxy_backup_servers{type=""server""}) by (host,
proxy)) by (proxy) >=
avg(sum(haproxy_active_servers{type=""backend""}) by (host, proxy) +
sum(haproxy_backup_servers{type=""backend""}) by (host, proxy)) by
(proxy)
- In 2019.2.11 and newer:
avg(sum(haproxy_active_servers{type="server"}) by (host, proxy) +
sum(haproxy_backup_servers{type="server"}) by (host, proxy)) by
(proxy) - avg(sum(haproxy_active_servers{type="backend"}) by (host,
proxy) + sum(haproxy_backup_servers{type="backend"}) by (host, proxy))
by (proxy) >= 0.5 * avg(sum(haproxy_active_servers{type="server"}) by
(host, proxy) + sum(haproxy_backup_servers{type="server"}) by (host,
proxy)) by (proxy)
|
|---|
| Description |
Raises when at least half of the back-end servers (>=50%) used by the
HAProxy service are in the DOWN state. The host and proxy
labels in the raised alert contain the host name of the affected node
and the service proxy name. |
|---|
| Troubleshooting |
- Inspect the HAProxy logs by running
journalctl -u haproxy on the
affected node.
- Verify the state of the affected back-end server:
- Verify that the server is responding and the back-end service is
active and responsive.
- Verify the state of the back-end service using an HTTP GET request,
for example,
curl -XGET http://ctl01:8888/. Typically, the
200 response code indicates the healthy state.
|
|---|
| Tuning |
Not required |
|---|
HaproxyBackendOutage
| Severity |
Critical |
|---|
| Summary |
All {{ $labels.proxy }} backends are down. |
|---|
| Raise condition |
max(haproxy_active_servers{sv=""BACKEND""}) by (proxy)
+ max(haproxy_backup_servers{sv=""BACKEND""}) by (proxy) == 0 |
|---|
| Description |
Raises when all back-end servers used by the HAProxy service across the
cluster are not available to process the requests proxied by HAProxy,
typically indicating deployment or configuration issues. The proxy
label in the raised alert contains the service proxy name. |
|---|
| Troubleshooting |
- Verify the affected backends.
- Inspect the HAProxy logs by running
journalctl -u haproxy on the
affected node.
- Inspect Telegraf logs by running
journalctl -u telegraf on the
affected node.
|
|---|
| Tuning |
Not required |
|---|