Restrict services to worker nodes

You can configure MKE to allow users to deploy and run services only in worker nodes. This ensures all cluster management functionality stays performant, and makes the cluster more secure.

Important

In the event that a user deploys a malicious service capable of affecting the node on which it is running, that service will not be able to strike any other nodes in the cluster or have any impact on cluster management functionality.

Swarm Workloads

To restrict users from deploying to manager nodes, log in with administrator credentials to the MKE web interface, navigate to the Admin Settings page, and choose Scheduler.

You can then choose if user services should be allowed to run on manager nodes or not.

Note

Creating a grant with the Scheduler role against the / collection takes precedence over any other grants with Node Schedule on subcollections.

Kubernetes Workloads

By default MKE clusters takes advantage of Taints and Tolerations to prevent a User’s workload being deployed on to MKE Manager or MSR Nodes.

You can view this taint by running:

$ kubectl get nodes <mkemanager> -o json | jq -r '.spec.taints | .[]'
{
  "effect": "NoSchedule",
  "key": "com.docker.ucp.manager"
}

Note

Workloads deployed by an Administrator in the kube-system namespace do not follow these scheduling constraints. If an Administrator deploys a workload in the kube-system namespace, a toleration is applied to bypass this taint, and the workload is scheduled on all node types.

Allow Administrators to Schedule on Manager / MSR Nodes

To allow Administrators to deploy workloads accross all nodes types, an Administrator can tick the “Allow administrators to deploy containers on MKE managers or nodes running MSR” box in the MKE web interface.

For all new workloads deployed by Administrators after this box has been ticked, MKE will apply a toleration to your workloads to allow the pods to be scheduled on all node types.

For existing workloads, the Administrator will need to edit the Pod specification, through kubectl edit <object> <workload> or the MKE web interface and add the following toleration:

tolerations:
- key: "com.docker.ucp.manager"
  operator: "Exists"

You can check this has been applied succesfully by:

$ kubectl get <object> <workload> -o json | jq -r '.spec.template.spec.tolerations | .[]'
{
  "key": "com.docker.ucp.manager",
  "operator": "Exists"
}

Allow Users and Service Accounts to Schedule on Manager / MSR Nodes

To allow Kubernetes Users and Service Accounts to deploy workloads accross all node types in your cluster, an Administrator will need to tick “Allow all authenticated users, including service accounts, to schedule on all nodes, including MKE managers and MSR nodes.” in the MKE web interface.

For all new workloads deployed by Kubernetes Users after this box has been ticked, MKE will apply a toleration to your workloads to allow the pods to be scheduled on all node types. For existing workloads, the User would need to edit Pod Specification as detailed above in the “Allow Administrators to Schedule on Manager / MSR Nodes” section.

There is a NoSchedule taint on MKE managers and MSR nodes and if you have scheduling on managers/workers disabled in the MKE scheduling options, then a toleration for that taint will not get applied to the deployments, so they should not schedule on those nodes. Unless the Kube workload is deployed in the kube-system name space.

See also

Kubernetes