You can configure MKE to allow users to deploy and run services only in worker nodes. This ensures all cluster management functionality stays performant, and makes the cluster more secure.
Important
In the event that a user deploys a malicious service capable of affecting the node on which it is running, that service will not be able to strike any other nodes in the cluster or have any impact on cluster management functionality.
To restrict users from deploying to manager nodes, log in with administrator credentials to the MKE web interface, navigate to the Admin Settings page, and choose Scheduler.
You can then choose if user services should be allowed to run on manager nodes or not.
Note
Creating a grant with the Scheduler role against the / collection
takes precedence over any other grants with Node Schedule on
subcollections.
By default MKE clusters takes advantage of Taints and Tolerations to prevent a User’s workload being deployed on to MKE Manager or MSR Nodes.
You can view this taint by running:
$ kubectl get nodes <ucpmanager> -o json | jq -r '.spec.taints | .[]'
{
"effect": "NoSchedule",
"key": "com.docker.ucp.manager"
}
Note
Workloads deployed by an Administrator in the kube-system
namespace do not follow these scheduling constraints. If an
Administrator deploys a workload in the kube-system namespace, a
toleration is applied to bypass this taint, and the workload is
scheduled on all node types.
To allow Administrators to deploy workloads accross all nodes types, an Administrator can tick the “Allow administrators to deploy containers on MKE managers or nodes running MSR” box in the MKE web interface.
For all new workloads deployed by Administrators after this box has been ticked, MKE will apply a toleration to your workloads to allow the pods to be scheduled on all node types.
For existing workloads, the Administrator will need to edit the Pod
specification, through kubectl edit <object> <workload> or the MKE
web interface and add the following toleration:
tolerations:
- key: "com.docker.ucp.manager"
operator: "Exists"
You can check this has been applied succesfully by:
$ kubectl get <object> <workload> -o json | jq -r '.spec.template.spec.tolerations | .[]'
{
"key": "com.docker.ucp.manager",
"operator": "Exists"
}
To allow Kubernetes Users and Service Accounts to deploy workloads accross all node types in your cluster, an Administrator will need to tick “Allow all authenticated users, including service accounts, to schedule on all nodes, including MKE managers and MSR nodes.” in the MKE web interface.
For all new workloads deployed by Kubernetes Users after this box has been ticked, MKE will apply a toleration to your workloads to allow the pods to be scheduled on all node types. For existing workloads, the User would need to edit Pod Specification as detailed above in the “Allow Administrators to Schedule on Manager / MSR Nodes” section.
There is a NoSchedule taint on MKE managers and MSR nodes and if you
have scheduling on managers/workers disabled in the MKE scheduling
options, then a toleration for that taint will not get applied to the
deployments, so they should not schedule on those nodes. Unless the Kube
workload is deployed in the kube-system name space.