In correlation with the end of life (EOL) for MKE 3.5.x, maintenance of this documentation set was discontinued as of 2023-NOV-22. Click here for the latest MKE 3.x version documentation.

Configure service clusters

This topic and the next assume that the following prerequisites have been met:

  • You have an operational MKE cluster with at least two worker nodes (mke-node-0 and mke-node-1), which you will use as dedicated proxy servers for two independent Interlock service clusters.

  • You have enabled Interlock with an HTTP port of 80 and an HTTPS port of 8443.


  1. From a manager node, apply node labels to the MKE workers that you have chosen to use as your proxy servers:

    docker node update --label-add nodetype=loadbalancer --label-add region=east mke-node-0
docker node update --label-add nodetype=loadbalancer --label-add region=west mke-node-1

    In this example, mke-node-0 serves as the proxy for the east region and mke-node-1 serves as the proxy for the west region.

  2. Create a dedicated overlay network for each region proxy to manage traffic:

    docker network create --driver overlay eastnet
docker network create --driver overlay westnet

  3. Modify the Interlock configuration to create two service clusters:

    CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ \
(index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' \
ucp-interlock)
docker config inspect --format '{{ printf "%s" .Spec.Data }}' \
$CURRENT_CONFIG_NAME > old_config.toml

  4. Create the following config.toml file that declares two service clusters, east and west:

    ListenAddr = ":8080"
DockerURL = "unix:///var/run/docker.sock"
AllowInsecure = false
PollInterval = "3s"

[Extensions]
  [Extensions.east]
    Image = "mirantis/ucp-interlock-extension:3.2.3"
    ServiceName = "ucp-interlock-extension-east"
    Args = []
    Constraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux"]
    ConfigImage = "mirantis/ucp-interlock-config:3.2.3"
    ConfigServiceName = "ucp-interlock-config-east"
    ProxyImage = "mirantis/ucp-interlock-proxy:3.2.3"
    ProxyServiceName = "ucp-interlock-proxy-east"
    ServiceCluster="east"
    Networks=["eastnet"]
    ProxyConfigPath = "/etc/nginx/nginx.conf"
    ProxyReplicas = 1
    ProxyStopSignal = "SIGQUIT"
    ProxyStopGracePeriod = "5s"
    ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.region==east"]
    PublishMode = "host"
    PublishedPort = 80
    TargetPort = 80
    PublishedSSLPort = 8443
    TargetSSLPort = 443
    [Extensions.east.Labels]
      "ext_region" = "east"
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.east.ContainerLabels]
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.east.ProxyLabels]
      "proxy_region" = "east"
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.east.ProxyContainerLabels]
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.east.Config]
      Version = ""
      HTTPVersion = "1.1"
      User = "nginx"
      PidPath = "/var/run/proxy.pid"
      MaxConnections = 1024
      ConnectTimeout = 5
      SendTimeout = 600
      ReadTimeout = 600
      IPHash = false
      AdminUser = ""
      AdminPass = ""
      SSLOpts = ""
      SSLDefaultDHParam = 1024
      SSLDefaultDHParamPath = ""
      SSLVerify = "required"
      WorkerProcesses = 1
      RLimitNoFile = 65535
      SSLCiphers = "HIGH:!aNULL:!MD5"
      SSLProtocols = "TLSv1.2"
      AccessLogPath = "/dev/stdout"
      ErrorLogPath = "/dev/stdout"
      MainLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" '\n\t\t    '$status $body_bytes_sent \"$http_referer\" '\n\t\t    '\"$http_user_agent\" \"$http_x_forwarded_for\"';"
      TraceLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" $status '\n\t\t    '$body_bytes_sent \"$http_referer\" \"$http_user_agent\" '\n\t\t    '\"$http_x_forwarded_for\" $reqid $msec $request_time '\n\t\t    '$upstream_connect_time $upstream_header_time $upstream_response_time';"
      KeepaliveTimeout = "75s"
      ClientMaxBodySize = "32m"
      ClientBodyBufferSize = "8k"
      ClientHeaderBufferSize = "1k"
      LargeClientHeaderBuffers = "4 8k"
      ClientBodyTimeout = "60s"
      UnderscoresInHeaders = false
      UpstreamZoneSize = 64
      ServerNamesHashBucketSize = 128
      GlobalOptions = []
      HTTPOptions = []
      TCPOptions = []
      HideInfoHeaders = false

  [Extensions.west]
    Image = "mirantis/ucp-interlock-extension:3.2.3"
    ServiceName = "ucp-interlock-extension-west"
    Args = []
    Constraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux"]
    ConfigImage = "mirantis/ucp-interlock-config:3.2.3"
    ConfigServiceName = "ucp-interlock-config-west"
    ProxyImage = "mirantis/ucp-interlock-proxy:3.2.3"
    ProxyServiceName = "ucp-interlock-proxy-west"
    ServiceCluster="west"
    Networks=["westnet"]
    ProxyConfigPath = "/etc/nginx/nginx.conf"
    ProxyReplicas = 1
    ProxyStopSignal = "SIGQUIT"
    ProxyStopGracePeriod = "5s"
    ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.region==west"]
    PublishMode = "host"
    PublishedPort = 80
    TargetPort = 80
    PublishedSSLPort = 8443
    TargetSSLPort = 443
    [Extensions.west.Labels]
      "ext_region" = "west"
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.west.ContainerLabels]
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.west.ProxyLabels]
      "proxy_region" = "west"
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.west.ProxyContainerLabels]
      "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
    [Extensions.west.Config]
      Version = ""
      HTTPVersion = "1.1"
      User = "nginx"
      PidPath = "/var/run/proxy.pid"
      MaxConnections = 1024
      ConnectTimeout = 5
      SendTimeout = 600
      ReadTimeout = 600
      IPHash = false
      AdminUser = ""
      AdminPass = ""
      SSLOpts = ""
      SSLDefaultDHParam = 1024
      SSLDefaultDHParamPath = ""
      SSLVerify = "required"
      WorkerProcesses = 1
      RLimitNoFile = 65535
      SSLCiphers = "HIGH:!aNULL:!MD5"
      SSLProtocols = "TLSv1.2"
      AccessLogPath = "/dev/stdout"
      ErrorLogPath = "/dev/stdout"
      MainLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" '\n\t\t    '$status $body_bytes_sent \"$http_referer\" '\n\t\t    '\"$http_user_agent\" \"$http_x_forwarded_for\"';"
      TraceLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" $status '\n\t\t    '$body_bytes_sent \"$http_referer\" \"$http_user_agent\" '\n\t\t    '\"$http_x_forwarded_for\" $reqid $msec $request_time '\n\t\t    '$upstream_connect_time $upstream_header_time $upstream_response_time';"
      KeepaliveTimeout = "75s"
      ClientMaxBodySize = "32m"
      ClientBodyBufferSize = "8k"
      ClientHeaderBufferSize = "1k"
      LargeClientHeaderBuffers = "4 8k"
      ClientBodyTimeout = "60s"
      UnderscoresInHeaders = false
      UpstreamZoneSize = 64
      ServerNamesHashBucketSize = 128
      GlobalOptions = []
      HTTPOptions = []
      TCPOptions = []
      HideInfoHeaders = false

    Note

    Change all instances of the MKE version and *.ucp.InstanceID in the above to match your deployment.

  5. Optional. Modify the configuration file that Interlock creates by default:

    1. Replace [Extensions.default] with [Extensions.east].

    2. Change ServiceName to "ucp-interlock-extension-east".

    3. Change ConfigServiceName to "ucp-interlock-config-east".

    4. Change ProxyServiceName to "ucp-interlock-proxy-east".

    5. Add the "node.labels.region==east" constraint to the ProxyConstraints list.

    6. Add the ServiceCluster="east" key immediately below and inline with ProxyServiceName.

    7. Add the Networks=["eastnet"] key immediately below and inline with ServiceCluster. This list can contain as many overlay networks as you require. Interlock only connects to the specified networks and connects to them all at startup.

    8. Change PublishMode="ingress" to PublishMode="host".

    9. Change the [Extensions.default.Labels] section title to [Extensions.east.Labels].

    10. Add the "ext_region" = "east" key under the [Extensions.east.Labels] section.

    11. Change the [Extensions.default.ContainerLabels] section title to [Extensions.east.ContainerLabels].

    12. Change the [Extensions.default.ProxyLabels] section title to [Extensions.east.ProxyLabels].

    13. Add the "proxy_region" = "east" key under the [Extensions.east.ProxyLabels] section.

    14. Change the [Extensions.default.ProxyContainerLabels] section title to [Extensions.east.ProxyContainerLabels].

    15. Change the [Extensions.default.Config] section title to [Extensions.east.Config].

    16. Optional. Change ProxyReplicas=2 to ProxyReplicas=1. This is only necessary if there is a single node labeled as a proxy for each service cluster.

    17. Configure your west service cluster by duplicating the entire [Extensions.east] block and changing all instances of east to west.

  6. Create a new docker config object from the config.toml file:

    NEW_CONFIG_NAME="com.docker.ucp.interlock.conf-$(( \
$(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME") + 1 ))"
docker config create $NEW_CONFIG_NAME config.toml

  7. Update the ucp-interlock service to start using the new configuration:

    docker service update \
--config-rm $CURRENT_CONFIG_NAME \
--config-add source=$NEW_CONFIG_NAME,target=/config.toml \
ucp-interlock

  8. View your service clusters:

    docker service ls

    The following two proxy services will display: ucp-interlock-proxy-east and ucp-interlock-proxy-west.

    Note

    If only one proxy service displays, delete it using docker service rm and rerun docker service ls to display the two new proxy services.