Configure service clusters

This topic and the next assume that the following prerequisites have been met:

  • You have an operational MKE cluster with at least two worker nodes (mke-node-0 and mke-node-1), which you will use as dedicated proxy servers for two independent Interlock service clusters.

  • You have enabled Interlock with an HTTP port of 80 and an HTTPS port of 8443.


  1. From a manager node, apply node labels to the MKE workers that you have chosen to use as your proxy servers:

    docker node update --label-add nodetype=loadbalancer --label-add region=east mke-node-0
    docker node update --label-add nodetype=loadbalancer --label-add region=west mke-node-1
    

    In this example, mke-node-0 serves as the proxy for the east region and mke-node-1 serves as the proxy for the west region.

  2. Create a dedicated overlay network for each region proxy to manage traffic:

    docker network create --driver overlay eastnet
    docker network create --driver overlay westnet
    
  3. Modify the Interlock configuration to create two service clusters:

    CURRENT_CONFIG_NAME=$(docker service inspect --format '{{ \
    (index .Spec.TaskTemplate.ContainerSpec.Configs 0).ConfigName }}' \
    ucp-interlock)
    docker config inspect --format '{{ printf "%s" .Spec.Data }}' \
    $CURRENT_CONFIG_NAME > old_config.toml
    
  4. Create the following config.toml file that declares two service clusters, east and west:

    ListenAddr = ":8080"
    DockerURL = "unix:///var/run/docker.sock"
    AllowInsecure = false
    PollInterval = "3s"
    
    [Extensions]
      [Extensions.east]
        Image = "mirantis/ucp-interlock-extension:3.2.3"
        ServiceName = "ucp-interlock-extension-east"
        Args = []
        Constraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux"]
        ConfigImage = "mirantis/ucp-interlock-config:3.2.3"
        ConfigServiceName = "ucp-interlock-config-east"
        ProxyImage = "mirantis/ucp-interlock-proxy:3.2.3"
        ProxyServiceName = "ucp-interlock-proxy-east"
        ServiceCluster="east"
        Networks=["eastnet"]
        ProxyConfigPath = "/etc/nginx/nginx.conf"
        ProxyReplicas = 1
        ProxyStopSignal = "SIGQUIT"
        ProxyStopGracePeriod = "5s"
        ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.region==east"]
        PublishMode = "host"
        PublishedPort = 80
        TargetPort = 80
        PublishedSSLPort = 8443
        TargetSSLPort = 443
        [Extensions.east.Labels]
          "ext_region" = "east"
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.east.ContainerLabels]
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.east.ProxyLabels]
          "proxy_region" = "east"
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.east.ProxyContainerLabels]
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.east.Config]
          Version = ""
          HTTPVersion = "1.1"
          User = "nginx"
          PidPath = "/var/run/proxy.pid"
          MaxConnections = 1024
          ConnectTimeout = 5
          SendTimeout = 600
          ReadTimeout = 600
          IPHash = false
          AdminUser = ""
          AdminPass = ""
          SSLOpts = ""
          SSLDefaultDHParam = 1024
          SSLDefaultDHParamPath = ""
          SSLVerify = "required"
          WorkerProcesses = 1
          RLimitNoFile = 65535
          SSLCiphers = "HIGH:!aNULL:!MD5"
          SSLProtocols = "TLSv1.2"
          AccessLogPath = "/dev/stdout"
          ErrorLogPath = "/dev/stdout"
          MainLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" '\n\t\t    '$status $body_bytes_sent \"$http_referer\" '\n\t\t    '\"$http_user_agent\" \"$http_x_forwarded_for\"';"
          TraceLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" $status '\n\t\t    '$body_bytes_sent \"$http_referer\" \"$http_user_agent\" '\n\t\t    '\"$http_x_forwarded_for\" $reqid $msec $request_time '\n\t\t    '$upstream_connect_time $upstream_header_time $upstream_response_time';"
          KeepaliveTimeout = "75s"
          ClientMaxBodySize = "32m"
          ClientBodyBufferSize = "8k"
          ClientHeaderBufferSize = "1k"
          LargeClientHeaderBuffers = "4 8k"
          ClientBodyTimeout = "60s"
          UnderscoresInHeaders = false
          UpstreamZoneSize = 64
          ServerNamesHashBucketSize = 128
          GlobalOptions = []
          HTTPOptions = []
          TCPOptions = []
          HideInfoHeaders = false
    
      [Extensions.west]
        Image = "mirantis/ucp-interlock-extension:3.2.3"
        ServiceName = "ucp-interlock-extension-west"
        Args = []
        Constraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux"]
        ConfigImage = "mirantis/ucp-interlock-config:3.2.3"
        ConfigServiceName = "ucp-interlock-config-west"
        ProxyImage = "mirantis/ucp-interlock-proxy:3.2.3"
        ProxyServiceName = "ucp-interlock-proxy-west"
        ServiceCluster="west"
        Networks=["westnet"]
        ProxyConfigPath = "/etc/nginx/nginx.conf"
        ProxyReplicas = 1
        ProxyStopSignal = "SIGQUIT"
        ProxyStopGracePeriod = "5s"
        ProxyConstraints = ["node.labels.com.docker.ucp.orchestrator.swarm==true", "node.platform.os==linux", "node.labels.region==west"]
        PublishMode = "host"
        PublishedPort = 80
        TargetPort = 80
        PublishedSSLPort = 8443
        TargetSSLPort = 443
        [Extensions.west.Labels]
          "ext_region" = "west"
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.west.ContainerLabels]
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.west.ProxyLabels]
          "proxy_region" = "west"
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.west.ProxyContainerLabels]
          "com.docker.ucp.InstanceID" = "vl5umu06ryluu66uzjcv5h1bo"
        [Extensions.west.Config]
          Version = ""
          HTTPVersion = "1.1"
          User = "nginx"
          PidPath = "/var/run/proxy.pid"
          MaxConnections = 1024
          ConnectTimeout = 5
          SendTimeout = 600
          ReadTimeout = 600
          IPHash = false
          AdminUser = ""
          AdminPass = ""
          SSLOpts = ""
          SSLDefaultDHParam = 1024
          SSLDefaultDHParamPath = ""
          SSLVerify = "required"
          WorkerProcesses = 1
          RLimitNoFile = 65535
          SSLCiphers = "HIGH:!aNULL:!MD5"
          SSLProtocols = "TLSv1.2"
          AccessLogPath = "/dev/stdout"
          ErrorLogPath = "/dev/stdout"
          MainLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" '\n\t\t    '$status $body_bytes_sent \"$http_referer\" '\n\t\t    '\"$http_user_agent\" \"$http_x_forwarded_for\"';"
          TraceLogFormat = "'$remote_addr - $remote_user [$time_local] \"$request\" $status '\n\t\t    '$body_bytes_sent \"$http_referer\" \"$http_user_agent\" '\n\t\t    '\"$http_x_forwarded_for\" $reqid $msec $request_time '\n\t\t    '$upstream_connect_time $upstream_header_time $upstream_response_time';"
          KeepaliveTimeout = "75s"
          ClientMaxBodySize = "32m"
          ClientBodyBufferSize = "8k"
          ClientHeaderBufferSize = "1k"
          LargeClientHeaderBuffers = "4 8k"
          ClientBodyTimeout = "60s"
          UnderscoresInHeaders = false
          UpstreamZoneSize = 64
          ServerNamesHashBucketSize = 128
          GlobalOptions = []
          HTTPOptions = []
          TCPOptions = []
          HideInfoHeaders = false
    

    Note

    Change all instances of the MKE version and *.ucp.InstanceID in the above to match your deployment.

  5. Optional. Modify the configuration file that Interlock creates by default:

    1. Replace [Extensions.default] with [Extensions.east].

    2. Change ServiceName to "ucp-interlock-extension-east".

    3. Change ConfigServiceName to "ucp-interlock-config-east".

    4. Change ProxyServiceName to "ucp-interlock-proxy-east".

    5. Add the "node.labels.region==east" constraint to the ProxyConstraints list.

    6. Add the ServiceCluster="east" key immediately below and inline with ProxyServiceName.

    7. Add the Networks=["eastnet"] key immediately below and inline with ServiceCluster. This list can contain as many overlay networks as you require. Interlock only connects to the specified networks and connects to them all at startup.

    8. Change PublishMode="ingress" to PublishMode="host".

    9. Change the [Extensions.default.Labels] section title to [Extensions.east.Labels].

    10. Add the "ext_region" = "east" key under the [Extensions.east.Labels] section.

    11. Change the [Extensions.default.ContainerLabels] section title to [Extensions.east.ContainerLabels].

    12. Change the [Extensions.default.ProxyLabels] section title to [Extensions.east.ProxyLabels].

    13. Add the "proxy_region" = "east" key under the [Extensions.east.ProxyLabels] section.

    14. Change the [Extensions.default.ProxyContainerLabels] section title to [Extensions.east.ProxyContainerLabels].

    15. Change the [Extensions.default.Config] section title to [Extensions.east.Config].

    16. Optional. Change ProxyReplicas=2 to ProxyReplicas=1. This is only necessary if there is a single node labeled as a proxy for each service cluster.

    17. Configure your west service cluster by duplicating the entire [Extensions.east] block and changing all instances of east to west.

  6. Create a new docker config object from the config.toml file:

    NEW_CONFIG_NAME="com.docker.ucp.interlock.conf-$(( \
    $(cut -d '-' -f 2 <<< "$CURRENT_CONFIG_NAME") + 1 ))"
    docker config create $NEW_CONFIG_NAME config.toml
    
  7. Update the ucp-interlock service to start using the new configuration:

    docker service update \
    --config-rm $CURRENT_CONFIG_NAME \
    --config-add source=$NEW_CONFIG_NAME,target=/config.toml \
    ucp-interlock
    
  8. View your service clusters:

    docker service ls
    

    The following two proxy services will display: ucp-interlock-proxy-east and ucp-interlock-proxy-west.

    Note

    If only one proxy service displays, delete it using docker service rm and rerun docker service ls to display the two new proxy services.