Mirantis Kubernetes Engine Service Discovery and Load Balancing for Kubernetes¶

Service Discovery with Environment Variables¶

When a pod gets spun up, the kubelet injects a set of environment variables into it. The name of all configured services and their associated ports within the same namespace are injected in every pod. These environment variables can then be used by the application to connect to these services. Below is an example of the env output to a pod. Note that there is a SQL Server service deployed within the same namespace.

$ env
app=dockerdemo
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
SQLSERVER_SERVICE_PORT=1433
SQLSERVER_PORT_1433_TCP=tcp://10.96.221.251:1433
SQLSERVER_PORT=tcp://10.96.221.251:1433
SQLSERVER_SERVICE_HOST=10.96.221.251

Service Discovery with DNS¶

DKS comes with a highly-available DNS service using kube-dns. DKS will be moving to CoreDNS in the next major release given that it’s a more performant and CNCF-graduated project.

Like any other service, the kube-dns service is also assigned a VIP. By default it’s 10.96.0.10 and gets injected into every pod’s /etc/resolv.conf as a nameserver by the node’s kubelet. Please see below:

$ kubectl get svc kube-dns -n kube-system -o wide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   30d   k8s-app=kube-dns

# FROM INSIDE THE POD
$ cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5

By default, all services are assigned a DNS A record for a name of the form <SERVICE-NAME>.<NAMESPACE>.svc.cluster.local. This record will be associated with a Virtual IP (VIP) for the service. Any other applications within the cluster, across all namespaces, should be able to resolve this service using its DNS name. If the application pod trying to resolve this service also lives within the same namespace (e.g default namespace) then it should be able to resolve it using <SERVICE-NAME> because the name resolution default to .<NAMESPACE>.svc.cluster.local. Otherwise, the namespace needs to be included in the DNS request.

More info on Kubernetes DNS for Services and Pods is here

ClusterIP¶

Services will receive a cluster-scoped Virtual IP address also known as ClusterIP. This IP is reserved from a separate subnet dedicated for services IP allocation(service-cluster-ip-range). This IP will be reachable from all namespaces within the cluster. This is the default method of exposing the service internally. Once the service is created and a VIP is associated with it, every kube-proxy running on every cluster node will program an iptables rule so that all traffic destined to that VIP will be redirected to one of the Service’s backend pods instead.

By default, and unless otherwise configured, the services subnet is 10.96.0.0/16 and the pods subnet is 192.168.0.0/16. These configurations can be customized using the --service-cluster-ip-range and --pod-cidr install options, respectively, only during the initial provisioning of the MKE cluster. More info on these customization can be found here.

Below is an example of a Service definition that uses ClusterIP.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  type: ClusterIP
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

If the service needs to be accessed from outside the cluster, then there are a couple of available options to doing so. Mainly NodePort and LoadBalancer. Ingress is another method that you can use, and we will discuss. Ingress is technically not a Type of a service, but it’s another method for using Layer 7 based routing to expose your services externally.

NodePort¶

If you use NodePort service type, a random and available TCP or UDP port will be allocated from the port range (default is “32768-35535” in DKS) of the node itself. Each node proxies that port (the same port number on every node) into your Service. If you want to specify the port that the node should listen on you can do that under the Ports section in the YAML config of the Service. If the port is already in-use, then the deployment will fail and you will see an error describing that. Here’s an example of a service with defined ports:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  type: NodePort
  ports:
  - name: http
    port: 8080
    targetPort: 80
    protocol: TCP

When this Service is deployed, all nodes in the DKS cluster will listen on port 8080 and forward to port 80 on the back-end pods.

This service type is well suited for application services that require direct TCP/UDP access, like RabbitMQ for example. If your service can be accessed using HTTP/HTTPs then Ingress is recommended (we will discuss Ingress in the following sections).

Final note, when you create a service with Type: NodePort, the service will still acquire a cluster IP to be used for internal cluster traffic.

LoadBalancer¶

If you are running Docker Enterprise on AWS or Azure, this service type allows you to integrate natively with the underlying cloud’s load balancing services by provisioning a load balancer for your Service and configuring it to forward traffic directly to your service. Although this can be useful in some cases, provisioning a dedicated external loadbalancer for your Kubernetes services can be an overkill. Below is an example of a service that utilizes AWS’s ELB integration:

$ cat service-lb.yaml
apiVersion: v1
kind: Service
metadata:
  name: docker-demo-svc-lb
spec:
  selector:
    app: dockerdemo
  ports:
    - port: 8080
      targetPort: 8080
  type: LoadBalancer

$ kubectl get svc
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)          AGE
docker-demo-svc-lb             LoadBalancer   10.96.0.69      XXXXXXXXX.us-west-2.elb.amazonaws.com   8080:35494/TCP   35m

More details on exposing services and service types is here

Option A: Istio-based Cluster Ingress (Experimental in MKE 3.2)¶

Cluster Ingress provides layer 7 services to traffic entering a Docker Enterprise cluster for a variety of different use-cases that help provide application resilience, security, and observability. Ingress offers dynamic control of L7 routing in a highly available architecture that is also highly performant. DKS’s Ingress for Kubernetes is based on the Istio control-plane and is a simplified deployment focused on just providing ingress services with minimal complexity, including features such as:

• L7 host and path routing
• Complex path matching and redirection rules
• Weight-based load balancing
• TLS termination
• Persistent L7 sessions
• Hot config reloads

Note: The next major release of MKE will include a built-in and supported version of this functionality out-of-the-box.

Full details on deploying this solution can be found here

Option B: NGINX Ingress Controller¶

NGINX Ingress controller is a widespread and production-ready solution that provides layer-7 routing functionality with ease. Although Docker does not officially support it, it had been tested to work with DKS. The following guide walks you through an end-to-end production deployment of this solution.

Once you deploy the NGINX Ingress controller, you can configure Ingress objects/routes for your application. For example, here is an ingress object docker-demo-svc service we described earlier:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dockerdemo-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: dockerdemo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: docker-demo-svc
          servicePort: 80

This specific example uses the NGINX ingress controller, indicate in the annotations section. The spec section is used to associate the L7 URL/host to the backend service.

Finally, once you deploy the ingress object, you can see your ingress routes under the Ingress section within MKE:

Detailed documentation on Ingress configurations can be found here