Mirantis Kubernetes Engine Service Discovery and Load Balancing for Swarm¶

Differences Between the Interlock Proxy and Swarm Mode Routing Mesh¶

The main difference between the Interlock Proxy and swarm mode routing mesh is that the Interlock Proxy is intended to be used only for HTTP traffic at the application layer, while the swarm mode routing mesh works at a lower level on the transport layer.

Deciding which to use depends on the application. If the application is intended to be publicly accessible and is an HTTP service, then the Interlock Proxy could be a good fit. If mutual TLS is required for the backend application, then using the transport layer would probably be preferred.

Another advantage of using the Interlock Proxy is that less configuration is required for traffic to be routed to the service. Often times only a DNS record is needed along with setting the label on the service. If a wildcard DNS entry is used, then no configuration outside of setting the service label is necessary. In many organizations, access to load balancers and DNS is restricted. Being able to control requests to applications by just a service label can empower developers to quickly iterate over changes. With the swarm mode routing mesh, any frontend load balancer can be configured to send traffic to the service’s published port.

The following diagram shows an example with wildcard DNS:

Enabling the Swarm Layer 7 Routing (Interlock Proxy)¶

The Interlock Proxy can be enabled from the MKE web console. To enable it:

1. Log into the MKE web console.
2. Navigate to Admin Settings > Layer 7 Routing.
3. Check Enable Layer 7 Routing under the section titled Swarm Layer 7 Routing (Interlock).
4. Configure the ports for Interlock Proxy to listen on, with the defaults being 8080 and 8443. The HTTPS port defaults to 8443 so that it doesn’t interfere with the default MKE management port (443).

Once enabled, MKE creates three services on the swarm cluster: ucp-interlock, ucp-interlock-extension, and ucp-interlock-proxy. The ucp-interlock-proxy service is responsible for routing traffic to the specified container based on the HTTP Host: header. Since the Interlock Proxy service is a swarm mode service, every node in the MKE cluster can route traffic to it by receiving traffic from ports 8080 and 8443. By default the Interlock Proxy service exposes ports 8080 and 8443 cluster-wide, and any requests on ports 8080 and 8443 to any node in the cluster are sent to the Interlock Proxy service.

Networks and Access Control¶

The Interlock Proxy uses one or more overlay networks to communicate with the backend application services. To allow the Interlock Proxy to communicate with and consequently forward requests to application frontend services, it needs to share a network with that service. This is accomplished by setting a label com.docker.lb.network to a value which is the name of the network the Interlock Proxy service should attach to for upstream connectivity. As such this action does not require administrator level access within MKE to be performed.

This configuration also allows the isolation between frontend services using the Interlock Proxy since the exposed application services do not share a common network with other similar exposed services.

Swarm Layer 7 Routing (Interlock Proxy) Requirements¶

There are three requirements that services must satisfy to use the Interlock Proxy:

1. The service must be connected to a network which is also defined as the value for the service label com.docker.lb.network.
2. The service must listen on a port. This port need not be exposed to the outside, but it should be configured as the value of the label com.docker.lb.port in the service.
3. The service must define a service label com.docker.lb.hosts to specify the host (or FQDN) served by the service. Multiple hosts can optionally be configured using a comma-separated list.

Configuring DNS with the Swarm Layer 7 Routing (Interlock Proxy)¶

This section covers how to configure DNS for services using the Interlock Proxy. To use the Interlock Proxy, a DNS record for the service needs to point to the MKE cluster. This can be accomplished through a variety of different ways because of the flexibility that the swarm mode routing mesh provides.

If a service needs to be publicly accessible for requests to foo.example.com, then the DNS record for that service can be configured in one of the following ways:

1. Configure DNS to point to any single node on the MKE cluster. All requests for foo.example.com will get routed through that node to the Interlock Proxy.
2. Configure round-robin DNS to point to multiple nodes on the MKE cluster. Any node that receives a request for foo.example.com will get routed through the Interlock Proxy.
3. Or, the best solution for high availability, is to configure an external HA load balancer to reside in front of the MKE cluster. There are some considerations to keep in mind when using an external HA load balancer:
   • Set the DNS record for foo.example.com to point to the external load balancer.
   • The external load balancer should point to multiple MKE nodes that reside in different availability zones for increased resiliency.
   • Configure the external load balancer to perform a TCP health check on the Interlock Proxy service’s configured exposed port(s) so that traffic will route through healthy MKE nodes.

Which nodes should be used to route traffic, managers, or workers? There are a few ways to approach that question.

1. Routing through the manager nodes is fine for smaller deployments since managers are generally more static in nature.
   • Advantage: Manager nodes generally do not shift around (new hosts, new IPs, etc.) often, and it is easier to keep load balancers pointing to the same nodes.
   • Disadvantage: Manager nodes are responsible for the control plane traffic. If application traffic is large, you don’t want to saturate traffic to these nodes and cause adverse affects to your cluster.
2. Routing through the worker nodes.
   • Advantage: Worker nodes do not manage the entire cluster, so there’s less additional networking overhead.
   • Disadvantage: Worker nodes are treated like “pets” instances, which can make infrastructure less repeatable and less reliable. Any automation built around destroying and building nodes needs to take this into account if load balancers are pointing to worker nodes.
3. Routing through specific worker nodes (Host mode). This setup is also referred to as Service clusters. In this setup, specific worker nodes will route traffic for a predetermined set of applications. Consequently, another set of worker nodes can be setup to route traffic for a different set of applications, potentially using a different configuration or extension.
   • Advantages:
     • Application traffic is completely segregated from one another.
     • Flexibility exists when applying different extensions / configurations for each application or application cluster.
     • Higher control exists over which hosts will route traffic to specific sets of applications or services. For instance, this feature can be leveraged to constrain the Interlock Proxy on hosts that are in different regions to offer high availability.
     • Updates or changes to the Interlock Proxy can be better planned such that it does not impact all applications / services. In other words, an outage to a service cluster will be localized to a failure of applications only within that service cluster.
   • Disadvantages: Due to managing the proxy hosts as fixed (host mode, without ingress), there is a cost around slight reduction in reliability for greater flexibility. If using ingress, then you will need to manage the ports corresponding to each service cluster. In both the cases, there is a small overhead of managing the configurations in this mode of routing.

Regardless of which type of instance your frontend load balancer is directing traffic to, it’s important to make sure the instances have an adequate network connection.

Routing in Interlock Proxy¶

For services to be published using Interlock Proxy, they must contain, among other labels, at least two labels where the keys are com.docker.lb.hosts and com.docker.lb.port.

• The value of the label com.docker.lb.hosts should be the host that the service should serve. Optionally it can also be a comma-separated list of the hosts that the service should serve.
• The value of the label com.docker.lb.port should contain the port to use for the internal upstream communication with the service. Note that this port on the service need not be published externally.
• Optionally, the label com.docker.lb.network can be set to point to the name of the network that Interlock Proxy service needs to attach to for upstream connectivity. This label is required only if the service to be published using Interlock Proxy is attached to multiple overlay networks.

Monitoring¶

To monitor the Interlock Proxy from a frontend load balancer, set the load balancer to monitor the exposed Interlock Proxy ports on the cluster using a TCP health check. If Interlock Proxy is configured to listen on the default ports of 8080 and 8443, then the frontend load balancer would need to simply perform a TCP health check on all nodes that are in its pool.

Interlock Proxy HA Considerations¶

This section discusses a few usage considerations with regards to Interlock Proxy running in high-availability mode.

The ucp-interlock-proxy can be scaled up to have more replicas, and those replicas can be constrained to only those nodes that have high performance network interfaces. The additional benefit of this architecture is that it improves security by avoiding all application traffic from routing through the managers.

The following steps needed to accomplish this design:

1. Update the high performance nodes by using node labels to identify them:

   $ docker node update --label-add nodetype=loadbalancer <node>
   

2. Constrain the Interlock Proxy service tasks to only run on the high performance nodes using the node labels. This is done by updating the ucp-interlock service configuration to deploy the interlock proxy service with the updated constraints in the ProxyConstraints array as explained below:

   • Retrieve the configuration that is currently being used for the ucp-interlock service and save it to a file (config.toml for example):

     $ CURRENT_CONFIG_NAME=$(docker service inspect --format \
         '{{ (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' \
         ucp-interlock)
     
     $ docker config inspect --format '{{ printf "%s" .Spec.Data }}' \
         $CURRENT_CONFIG_NAME > config.toml
     

   • Update the ProxyConstraints array in the config.toml file as shown below:

     [Extensions]
       [Extensions.default]
         ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.nodetype==loadbalancer"]
     

   • Create a new Docker config from the file that was just edited:

     $ docker config create $NEW_CONFIG_NAME config.toml
     

   • Update the ucp-interlock service to start using the new configuration:

     $ docker service update \
         --config-rm $CURRENT_CONFIG_NAME \
         --config-add source=$NEW_CONFIG_NAME,target=/config.toml \
         ucp-interlock
     

3. Configure the upstream load balancer to direct requests to only those high performance nodes on the Interlock Proxy ports. This ensures all traffic is directed to only these nodes.

For more information and steps see: - Configure Layer 7 routing for production - Configure Layer 7 routing service

Interlock Proxy Routing Example¶

Consider an example, standard 3-tier application that showcases service discovery and load balancing in Docker EE.

To deploy the application stack, run these commands with the MKE client bundle loaded:

$ wget https://raw.githubusercontent.com/dockersuccess/counter-demo/master/interlock-docker-compose.yml
$ DOMAIN=<domain-to-route> docker stack deploy -c interlock-docker-compose.yml http-example

Then access the example application at http://<domain-to-route>/.

Websockets¶

The example also demonstrates support for websockets using the label com.docker.lb.websocket_endpoints with its value set to /total as shown in this section. The value can also be a comma-separated list of endpoints to configure to be upgraded for websockets.

This is the contents of the compose file if you just want to copy/paste into the MKE UI instead:

version: "3.3"

services:
  web:
    image: dockersuccess/webserver:latest
    environment:
      app_url: app:8080
    deploy:
      replicas: 2
      labels:
        com.docker.lb.hosts: ${DOMAIN:-app.dockerdemos.com}
        com.docker.lb.port: 2015
        com.docker.lb.websocket_endpoints: /total
    networks:
      - frontend
  app:
    image: dockersuccess/counter-demo:latest
    environment:
      ENVIRONMENT: ${env:-PRODUCTION}
    deploy:
      replicas: 5
      endpoint_mode: dnsrr
    networks:
      - frontend
      - backend
  db:
    image: redis:latest
    volumes:
      - data:/data
    networks:
      backend:

networks:
  frontend:
    driver: overlay
  backend:
    driver: overlay

volumes:
  data:

It is also possible to deploy through the MKE UI by going to Shared Resources -> Stacks -> Create Stack. Name the stack, change the Mode to Swarm Services, and copy/paste the above compose file into the open text field. Be sure to replace ${DOMAIN:-app.dockerdemos.com} with the correct domain name when deploying through the UI. Click on the Create button to deploy the stack.

Interlock Proxy Sticky Session Example¶

# Create an overlay network so that service traffic is isolated and secure
$ docker network create -d overlay demo

# Next create the service with the cookie to use for sticky sessions.
#   Replace <domain-to-route> with a valid domain.
$ docker service create \
  --name demo \
  --network demo \
  --detach=false \
  --replicas=5 \
  --label com.docker.lb.hosts=<domain-to-route> \
  --label com.docker.lb.sticky_session_cookie=session \
  --label com.docker.lb.port=8080 \
  --env METADATA="demo-sticky" \
  dockersuccess/docker-demo

$ curl -vs -c cookie.txt -b cookie.txt http://demo.local/ping
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
GET /ping HTTP/1.1
Host: demo.local
User-Agent: curl/7.54.0
Accept: */*
Cookie: session=1510171444496686286

< HTTP/1.1 200 OK
< Server: nginx/1.13.6
< Date: Wed, 08 Nov 2017 20:04:36 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 117
< Connection: keep-alive
* Replaced cookie session="1510171444496686286" for domain demo.local, path /, expire 0
< Set-Cookie: session=1510171444496686286
< x-request-id: 3014728b429320f786728401a83246b8
< x-proxy-id: eae36bf0a3dc
< x-server-info: interlock/2.0.0 (147ff2b1) linux/amd64
< x-upstream-addr: 10.0.2.5:8080
< x-upstream-response-time: 1510171476.948
<
{"instance":"9c67a943ffce","version":"0.1","metadata":"demo-sticky","request_id":"3014728b429320f786728401a83246b8"}

$ docker network create -d overlay demo
1se1glh749q1i4pw0kf26mfx5

$ docker service create \
    --name demo \
    --network demo \
    --detach=false \
    --replicas=5 \
    --label com.docker.lb.hosts=demo.local \
    --label com.docker.lb.port=8080 \
    --label com.docker.lb.ip_hash=true \
    --env METADATA="demo-sticky" \
    dockersuccess/docker-demo

Interlock Proxy HTTPS Example¶

Interlock Proxy has support for routing using HTTPS / SSL. Both “SSL Termination” and “SSL Passthrough” can be setup to provide different configurations of load balancing encrypted web traffic.

In SSL Terminations, SSL encrypted requests are decrypted by Interlock Proxy (the Load Balancer layer), and the unencrypted request is sent to the backend servers.

In SSL Passthrough, the SSL requests are sent as-is directly to the backend servers. The requests remain encrypted, and the backend servers become responsible for the decryption. This also implies that the backend servers have the necessary certificates and libraries to perform the decryption.

Secrets and certificates are almost always involved when dealing with encrypted communications. Before deploying the example application, generate the necessary certificates. You can also use one of several Certificate Authorities like Let’s Encrypt to generate the certificates.

Here are some helpful commands to generate self-signed certificates to use with the application using openssl.

$ openssl req \
    -new \
    -newkey rsa:4096 \
    -days 3650 \
    -nodes \
    -x509 \
    -subj "/C=US/ST=SomeState/L=SomeCity/O=Interlock/CN=demo.local" \
    -keyout demo.local.key \
    -out demo.local.cert

Now that the certificates are generated, you can use the resulting files as input to create two Docker secrets using the following commands:

$ docker secret create demo.local.cert demo.local.cert
ywn8ykni6cmnq4iz64um1pj7s
$ docker secret create demo.local.key demo.local.key
e2xo036ukhfapip05c0sizf5w

The secrets are now encrypted in the cluster-wide key value store. The secrets are encrypted at rest and using TLS while in motion to nodes that need the secret. Secrets can only be viewed by the application that needs to use them.

Now create an overlay network so that service traffic is isolated and secure:

$ docker network create -d overlay demo
1se1glh749q1i4pw0kf26mfx5

$ docker service create \
    --name demo \
    --network demo \
    --label com.docker.lb.hosts=demo.local \
    --label com.docker.lb.port=8080 \
    --label com.docker.lb.ssl_cert=demo.local.cert \
    --label com.docker.lb.ssl_key=demo.local.key \
    dockersuccess/docker-demo
6r0wiglf5f3bdpcy6zesh1pzx

Interlock Proxy detects when the service is available and publish it. Once the tasks are running and the proxy service has been updated the application should be available via https://demo.local.

$ curl -vsk https://demo.local/ping
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to demo.local (127.0.0.1) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=SomeState; L=SomeCity; O=Interlock; CN=demo.local
*  start date: Nov  8 16:23:03 2017 GMT
*  expire date: Nov  6 16:23:03 2027 GMT
*  issuer: C=US; ST=SomeState; L=SomeCity; O=Interlock; CN=demo.local
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /ping HTTP/1.1
> Host: demo.local
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.13.6
< Date: Wed, 08 Nov 2017 16:26:55 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 92
< Connection: keep-alive
< Set-Cookie: session=1510158415298009207; Path=/; Expires=Thu, 09 Nov 2017 16:26:55 GMT; Max-Age=86400
< x-request-id: 4b15ab2aaf2e0bbdea31f5e4c6b79ebd
< x-proxy-id: a783b7e646af
< x-server-info: interlock/2.0.0 (147ff2b1) linux/amd64
< x-upstream-addr: 10.0.2.3:8080

{"instance":"c2f1afe673d4","version":"0.1",request_id":"7bcec438af14f8875ffc3deab9215bc5"}

Since the certificate and key are stored securely within the Docker swarm, you can safely scale this service as well as the proxy service, and the Docker swarm will handle granting access to the credentials only as needed.

Use Service Clusters for Interlock Segmentation¶

Interlock service clusters allow Interlock to be segmented into multiple logical instances called “service clusters”, which have independently managed proxies. Application traffic only uses the proxies for a specific service cluster, allowing the full segmentation of traffic. Each service cluster only connects to the networks using that specific service cluster, which reduces the number of overlay networks to which proxies connect. Because service clusters also deploy separate proxies, this also reduces the amount of churn in LB configs when there are service updates.

Minimizing Number of Overlay Networks¶

The Interlock proxy containers will connect to the overlay network of every Swarm service. Having many networks connected to Interlock adds incremental delay when Interlock updates its load balancer configuration. Each network connected to Interlock generally adds 1-2 seconds of update delay. With many networks the Interlock update delay will cause the LB config to be out of date for too long which can cause traffic to be dropped.

Minimizing the number of overlay networks that Interlock connects to can be accomplished in two ways: - Reduce the number of networks. If the architecture permits it, applications can be grouped together to use the same networks. - Use Interlock Service Clusters. By segmenting Interlock, service clusters also segment which networks are connected to Interlock, reducing the number of networks that each proxy is connected to.

Use Interlock VIP Mode¶

VIP Mode can be used to reduce the impact of application updates on the Interlock proxies. It utilizes the Swarm L4 load balancing VIPs instead of individual task IPs to load balance traffic to a more stable internal endpoint. This prevents the proxy LB configs from changing for most kinds of app service updates reducing churn for Interlock. The major tradeoff is that the following features are not supported in VIP mode: - Sticky Sessions - Canary Deployments

The following features are still supported in VIP mode: - Host & Context Routing - Context Root Rewrites - Interlock TLS Termination - TLS Passthrough - Service Clusters

For additional tips on optimizing interlock for applications, see Optimizing Interlock for applications