Configure passive health checks for upstream servers

Available as of MKE 3.8.16

Interlock configures the NGINX proxy to perform passive health checks on the upstream servers, which are the tasks that back your Swarm services. By default, when an upstream server fails to respond, NGINX temporarily marks it as unavailable and stops routing traffic to it.

This section describes how to tune the passive health checks for a service by using the com.docker.lb.max_fails and com.docker.lb.fail_timeout service labels. Interlock applies the label values to every upstream server of the service, mapping them directly to the max_fails and fail_timeout parameters of the NGINX server directive. For the label descriptions and default values, refer to Use service labels.

Because Interlock manages these settings, MKE persists them and reapplies them automatically across reconciliation, upgrade, rollback, and restore operations.

Warning

Do not edit nginx.conf manually in the ucp-interlock-proxy containers. MKE overwrites manual changes when it reconciles the configuration.

Health check tuning guidelines

NGINX tracks failed requests to each upstream server. When a server accumulates max_fails failures within the fail_timeout window, NGINX marks that server unavailable for the duration of fail_timeout.

For a service that runs multiple replicas and occasionally returns transient errors or times out under load, the default threshold can cause NGINX to temporarily disable individual replicas. If enough replicas are marked unavailable at once, requests fail with 502 Bad Gateway and NGINX logs no live upstreams, even though the service itself is healthy.

Adjust the max_fails and fail_timeout labels when the proxy intermittently removes a healthy, busy service from rotation:

  • Increase max_fails to require more failures before NGINX marks a replica unavailable.

  • Decrease fail_timeout to shorten both the window in which NGINX counts failures and the period for which NGINX excludes a replica.

Note

You cannot disable passive health checks by setting max_fails to 0. Interlock treats com.docker.lb.max_fails=0 the same as an unset label, and NGINX applies its default value of 1.

Set the health check labels

Set the max_fails and fail_timeout labels on the service, either at creation time or in a Compose file.

Caution

Set com.docker.lb.max_fails to a non-negative integer and com.docker.lb.fail_timeout to a valid NGINX time value. Interlock skips a service with an invalid value and does not add it to the proxy configuration.

To configure the labels with docker service create:

docker service create \
--name <service-name> \
--network <network-name> \
--label com.docker.lb.hosts=<hostname> \
--label com.docker.lb.port=<port> \
--label com.docker.lb.max_fails=<max-fails> \
--label com.docker.lb.fail_timeout=<fail-timeout> \
<image>

For example, to create a demo service that tolerates up to 1000 failures within a 2s window:

docker service create \
--name demo \
--network demo \
--label com.docker.lb.hosts=app.example.org \
--label com.docker.lb.port=8080 \
--label com.docker.lb.max_fails=1000 \
--label com.docker.lb.fail_timeout=2s \
mirantiseng/docker-demo

To configure the labels in a Compose file:

  1. In the Compose file, add the labels under the service deploy.labels key:

    version: "3.2"
services:
  <service-name>:
    image: <image>
    deploy:
      replicas: <replicas>
      labels:
        com.docker.lb.hosts: <hostname>
        com.docker.lb.network: <stack-name>_<network-name>
        com.docker.lb.port: <port>
        com.docker.lb.max_fails: <max-fails>
        com.docker.lb.fail_timeout: <fail-timeout>
    networks:
      - <network-name>
networks:
  <network-name>:
    driver: overlay

    For example, for a demo stack with three replicas that tolerate up to 1000 failures within a 2s window:

    version: "3.2"
services:
  demo:
    image: mirantiseng/docker-demo
    deploy:
      replicas: 3
      labels:
        com.docker.lb.hosts: app.example.org
        com.docker.lb.network: demo_demo-network
        com.docker.lb.port: 8080
        com.docker.lb.max_fails: 1000
        com.docker.lb.fail_timeout: 2s
    networks:
      - demo-network
networks:
  demo-network:
    driver: overlay

    Note

    When you deploy a stack, Docker prefixes the network name with the stack name. As such, the com.docker.lb.network label must reference <stack-name>_<network-name>.

  2. Deploy the stack:

    docker stack deploy -c <compose-file> <stack-name>

    For example:

    docker stack deploy -c demo.yml demo

Interlock detects the label change and reconfigures the proxy.

Verify the configuration

Confirm that Interlock applied the values by printing the rendered NGINX configuration from a proxy container:

  1. On a node that runs the ucp-interlock-proxy service, obtain the name of a proxy container:

    docker ps --filter name=ucp-interlock-proxy --format '{{.Names}}'

  2. Print the rendered NGINX configuration:

    docker exec <ucp-interlock-proxy-container> nginx -T

    The upstream block for your service lists max_fails and fail_timeout on each server line:

    upstream up-<hostname> {
    zone up-<hostname>_backend 64k;
    server <task-ip-1>:<port> max_fails=<max-fails> fail_timeout=<fail-timeout>;
    server <task-ip-2>:<port> max_fails=<max-fails> fail_timeout=<fail-timeout>;
    server <task-ip-3>:<port> max_fails=<max-fails> fail_timeout=<fail-timeout>;
}

    For example, for the three-replica demo service from the Compose file example:

    upstream up-app.example.org {
    zone up-app.example.org_backend 64k;
    server 10.0.2.3:8080 max_fails=1000 fail_timeout=2s;
    server 10.0.2.4:8080 max_fails=1000 fail_timeout=2s;
    server 10.0.2.5:8080 max_fails=1000 fail_timeout=2s;
}

    If you omit a label, the corresponding parameter does not appear on the server lines, and NGINX applies its default value.

To change the values, update the labels on the service:

  • For a service that you created with docker service create, run:

    docker service update \
--label-add com.docker.lb.max_fails=<max-fails> \
--label-add com.docker.lb.fail_timeout=<fail-timeout> \
<service-name>

  • For a stack, update the labels in the Compose file and run docker stack deploy -c <compose-file> <stack-name> again.

Interlock re-renders the configuration with the new values.