Configure service clusters

Note

The provided instruction is based on the presumption that certain prerequisites have been met:

• You have an operational MKE cluster with at least two worker nodes (mke-node-0 and mke-node-1), to use as dedicated proxy servers for two independent Interlock service clusters.

• You have enabled Interlock with 80 as an HTTP port and 8443 as an HTTPS port.

1. From a manager node, apply node labels to the MKE workers that you have chosen to use as your proxy servers:

   docker node update --label-add nodetype=loadbalancer --label-add region=east mke-node-0
   docker node update --label-add nodetype=loadbalancer --label-add region=west mke-node-1
   

   In this example, mke-node-0 serves as the proxy for the east region and mke-node-1 serves as the proxy for the west region.

2. Create a dedicated overlay network for each region proxy to manage traffic:

   docker network create --driver overlay eastnet
   docker network create --driver overlay westnet
   

3. Modify the Interlock configuration to create two service clusters:

   CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ \
   (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' \
   ucp-interlock)
   docker config inspect --format '{{ printf "%s" .Spec.Data }}' \
   $CURRENT_CONFIG_NAME > old_config.toml
   

4. Create the following config.toml file that declares two service clusters, east and west:

   ListenAddr = ":8080"
   DockerURL = "unix:///var/run/docker.sock"
   AllowInsecure = false
   PollInterval = "3s"
   
   [Extensions]
     [Extensions.east]
       Image = "mirantis/ucp-interlock-extension:3.2.3"
       ServiceName = "ucp-interlock-extension-east"
       Args = []
       Constraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux"]
       ConfigImage = "mirantis/ucp-interlock-config:3.2.3"
       ConfigServiceName = "ucp-interlock-config-east"
       ProxyImage = "mirantis/ucp-interlock-proxy:3.2.3"
       ProxyServiceName = "ucp-interlock-proxy-east"
       ServiceCluster="east"
       Networks=["eastnet"]
       ProxyConfigPath = "/etc/nginx/nginx.conf"
       ProxyReplicas = 1
       ProxyStopSignal = "SIGQUIT"
       ProxyStopGracePeriod = "5s"
       ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.region==east"]
       PublishMode = "host"
       PublishedPort = 80
       TargetPort = 80
       PublishedSSLPort = 8443
       TargetSSLPort = 443
       [Extensions.east.Labels]
         "ext_region" = "east"
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.east.ContainerLabels]
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.east.ProxyLabels]
         "proxy_region" = "east"
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.east.ProxyContainerLabels]
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.east.Config]
         Version = ""
         HTTPVersion = "1.1"
         User = "nginx"
         PidPath = "/var/run/proxy.pid"
         MaxConnections = 1024
         ConnectTimeout = 5
         SendTimeout = 600
         ReadTimeout = 600
         IPHash = false
         AdminUser = ""
         AdminPass = ""
         SSLOpts = ""
         SSLDefaultDHParam = 1024
         SSLDefaultDHParamPath = ""
         SSLVerify = "required"
         WorkerProcesses = 1
         RLimitNoFile = 65535
         SSLCiphers = "HIGH:!aNULL:!MD5"
         SSLProtocols = "TLSv1.2"
         AccessLogPath = "/dev/stdout"
         ErrorLogPath = "/dev/stdout"
         MainLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" '\n\t\t    '$status $body_bytes_sent \"$http_referer\" '\n\t\t    '\"$http_user_agent\" \"$http_x_forwarded_for\"';"
         TraceLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" $status '\n\t\t    '$body_bytes_sent \"$http_referer\" \"$http_user_agent\" '\n\t\t    '\"$http_x_forwarded_for\" $reqid $msec $request_time '\n\t\t    '$upstream_connect_time $upstream_header_time $upstream_response_time';"
         KeepaliveTimeout = "75s"
         ClientMaxBodySize = "32m"
         ClientBodyBufferSize = "8k"
         ClientHeaderBufferSize = "1k"
         LargeClientHeaderBuffers = "4 8k"
         ClientBodyTimeout = "60s"
         UnderscoresInHeaders = false
         UpstreamZoneSize = 64
         ServerNamesHashBucketSize = 128
         GlobalOptions = []
         HTTPOptions = []
         TCPOptions = []
         HideInfoHeaders = false
   
     [Extensions.west]
       Image = "mirantis/ucp-interlock-extension:3.2.3"
       ServiceName = "ucp-interlock-extension-west"
       Args = []
       Constraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux"]
       ConfigImage = "mirantis/ucp-interlock-config:3.2.3"
       ConfigServiceName = "ucp-interlock-config-west"
       ProxyImage = "mirantis/ucp-interlock-proxy:3.2.3"
       ProxyServiceName = "ucp-interlock-proxy-west"
       ServiceCluster="west"
       Networks=["westnet"]
       ProxyConfigPath = "/etc/nginx/nginx.conf"
       ProxyReplicas = 1
       ProxyStopSignal = "SIGQUIT"
       ProxyStopGracePeriod = "5s"
       ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.region==west"]
       PublishMode = "host"
       PublishedPort = 80
       TargetPort = 80
       PublishedSSLPort = 8443
       TargetSSLPort = 443
       [Extensions.west.Labels]
         "ext_region" = "west"
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.west.ContainerLabels]
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.west.ProxyLabels]
         "proxy_region" = "west"
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.west.ProxyContainerLabels]
         "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
       [Extensions.west.Config]
         Version = ""
         HTTPVersion = "1.1"
         User = "nginx"
         PidPath = "/var/run/proxy.pid"
         MaxConnections = 1024
         ConnectTimeout = 5
         SendTimeout = 600
         ReadTimeout = 600
         IPHash = false
         AdminUser = ""
         AdminPass = ""
         SSLOpts = ""
         SSLDefaultDHParam = 1024
         SSLDefaultDHParamPath = ""
         SSLVerify = "required"
         WorkerProcesses = 1
         RLimitNoFile = 65535
         SSLCiphers = "HIGH:!aNULL:!MD5"
         SSLProtocols = "TLSv1.2"
         AccessLogPath = "/dev/stdout"
         ErrorLogPath = "/dev/stdout"
         MainLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" '\n\t\t    '$status $body_bytes_sent \"$http_referer\" '\n\t\t    '\"$http_user_agent\" \"$http_x_forwarded_for\"';"
         TraceLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" $status '\n\t\t    '$body_bytes_sent \"$http_referer\" \"$http_user_agent\" '\n\t\t    '\"$http_x_forwarded_for\" $reqid $msec $request_time '\n\t\t    '$upstream_connect_time $upstream_header_time $upstream_response_time';"
         KeepaliveTimeout = "75s"
         ClientMaxBodySize = "32m"
         ClientBodyBufferSize = "8k"
         ClientHeaderBufferSize = "1k"
         LargeClientHeaderBuffers = "4 8k"
         ClientBodyTimeout = "60s"
         UnderscoresInHeaders = false
         UpstreamZoneSize = 64
         ServerNamesHashBucketSize = 128
         GlobalOptions = []
         HTTPOptions = []
         TCPOptions = []
         HideInfoHeaders = false
   

   Note

   Change all instances of the MKE version and *.ucp.InstanceID in the above to match your deployment.

5. Optional. Modify the configuration file that Interlock creates by default:

   1. Replace [Extensions.default] with [Extensions.east].

   2. Change ServiceName to "ucp-interlock-extension-east".

   3. Change ConfigServiceName to "ucp-interlock-config-east".

   4. Change ProxyServiceName to "ucp-interlock-proxy-east".

   5. Add the "node.labels.region==east" constraint to the ProxyConstraints list.

   6. Add the ServiceCluster="east" key immediately below and inline with ProxyServiceName.

   7. Add the Networks=["eastnet"] key immediately below and inline with ServiceCluster. This list can contain as many overlay networks as you require. Interlock only connects to the specified networks and connects to them all at startup.

   8. Change PublishMode="ingress" to PublishMode="host".

   9. Change the [Extensions.default.Labels] section title to [Extensions.east.Labels].

   10. Add the "ext_region" = "east" key under the [Extensions.east.Labels] section.

   11. Change the [Extensions.default.ContainerLabels] section title to [Extensions.east.ContainerLabels].

   12. Change the [Extensions.default.ProxyLabels] section title to [Extensions.east.ProxyLabels].

   13. Add the "proxy_region" = "east" key under the [Extensions.east.ProxyLabels] section.

   14. Change the [Extensions.default.ProxyContainerLabels] section title to [Extensions.east.ProxyContainerLabels].

   15. Change the [Extensions.default.Config] section title to [Extensions.east.Config].

   16. Optional. Change ProxyReplicas=2 to ProxyReplicas=1. This is only necessary if there is a single node labeled as a proxy for each service cluster.

   17. Configure your west service cluster by duplicating the entire [Extensions.east] block and changing all instances of east to west.

6. Create a new docker config object from the config.toml file:

   NEW_CONFIG_NAME="com.docker.ucp.interlock.conf-$(( \
   $(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME") + 1 ))"
   docker config create $NEW_CONFIG_NAME config.toml
   

7. Update the ucp-interlock service to start using the new configuration:

   docker service update \
   --config-rm $CURRENT_CONFIG_NAME \
   --config-add source=$NEW_CONFIG_NAME,target=/config.toml \
   ucp-interlock
   

8. View your service clusters:

   docker service ls
   

   The following two proxy services will display: ucp-interlock-proxy-east and ucp-interlock-proxy-west.

   Note

   If only one proxy service displays, delete it using docker service rm and rerun docker service ls to display the two new proxy services.