Pod Security Policies (PSPs) are cluster-level resources that are enabled by default in MKE 3.2.
There are two default PSPs in MKE: a privileged policy and an
unprivileged policy. Administrators of the cluster can enforce
additional policies and apply them to users and teams for further
control of what runs in the Kubernetes cluster. This topic describes the
two default policies, and provides two example use cases for custom
policies.
To interact with PSPs, a user will need to be granted access to the
PodSecurityPolicy object in Kubernetes RBAC. If the user is a
MKE Admin, then the user can already manipulate PSPs. A normal user
can interact with policies if a MKE admin creates the following
ClusterRole and ClusterRoleBinding:
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp-admin
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
verbs:
- create
- delete
- get
- list
- patch
- update
EOF
$ USER=jeff
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp-admin:$USER
roleRef:
kind: ClusterRole
name: psp-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: User
name: $USER
EOF
By default, there are two policies defined within MKE, privileged
and unprivileged. Additionally, there is a ClusterRoleBinding
that gives every single user access to the privileged policy. This is
for backward compatibility after an upgrade. By default, any user can
create any pod.
Note
PSPs do not override security defaults built into the MKE RBAC engine for Kubernetes pods. These security defaults prevent non-admin users from mounting host paths into pods or starting privileged pods.
$ kubectl get podsecuritypolicies
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
privileged true * RunAsAny RunAsAny RunAsAny RunAsAny false *
unprivileged false RunAsAny RunAsAny RunAsAny RunAsAny false *
Specification for the privileged policy:
allowPrivilegeEscalation: true
allowedCapabilities:
- '*'
fsGroup:
rule: RunAsAny
hostIPC: true
hostNetwork: true
hostPID: true
hostPorts:
- max: 65535
min: 0
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
Specification for the unprivileged policy:
allowPrivilegeEscalation: false
allowedHostPaths:
- pathPrefix: /dev/null
readOnly: true
fsGroup:
rule: RunAsAny
hostPorts:
- max: 65535
min: 0
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
Note
When following this guide, if the prompt $ follows
admin, the action needs to be performed by a user with access to
create pod security policies as discussed in the Kubernetes RBAC
section. If the prompt
$ follows user, the MKE account does not need access to the
PSP object in Kubernetes. The user only needs the ability to create
Kubernetes pods.
To switch users from the privileged policy to the unprivileged
policy (or any custom policy), an admin must first remove the
ClusterRoleBinding that links all users and service accounts to the
privileged policy.
admin $ kubectl delete clusterrolebindings ucp:all:privileged-psp-role
When the ClusterRoleBinding is removed, cluster admins can still
deploy pods, and these pods are deployed with the privileged policy.
But users or service accounts are unable to deploy pods, because
Kubernetes does not know what pod security policy to apply. Note that cluster
admins would not be able to deploy deployments.
user $ kubectl apply -f pod.yaml
Error from server (Forbidden): error when creating "pod.yaml": pods "demopod" is forbidden: unable to validate against any pod security policy: []
Therefore, to allow a user or a service account to use the
unprivileged policy (or any custom policy), you must create a
RoleBinding to link that user or team with the alternative policy.
For the unprivileged policy, a ClusterRole has already been
defined, but has not been attached to a user.
# List Existing Cluster Roles
admin $ kubectl get clusterrole | grep psp
privileged-psp-role 3h47m
unprivileged-psp-role 3h47m
# Define which user to apply the ClusterRole too
admin $ USER=jeff
# Create a RoleBinding linking the ClusterRole to the User
admin $ cat <<EOF | kubectl create -f -
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: unprivileged-psp-role:$USER
namespace: default
roleRef:
kind: ClusterRole
name: unprivileged-psp-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: User
name: $USER
namespace: default
EOF
In the following example, when user “jeff” deploys a basic nginx
pod, the unprivileged policy then gets applied.
user $ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: demopod
spec:
containers:
- name: demopod
image: nginx
EOF
user $ kubectl get pods
NAME READY STATUS RESTARTS AGE
demopod 1/1 Running 0 10m
To check which PSP is applied to a pod, you can get a detailed view of
the pod spec using the -o yaml or -o json syntax with
kubectl. You can parse JSON output with jq.
user $ kubectl get pods demopod -o json | jq -r '.metadata.annotations."kubernetes.io/psp"'
unprivileged
Note
In most use cases, a Pod is not actually scheduled by a user. When creating Kubernetes objects such as Deployments or Daemonsets, the pods are scheduled by a service account or a controller.
If you have disabled the privileged PSP policy, and created a
RoleBinding to map a user to a new PSP policy, Kubernetes objects
like Deployments and Daemonsets will not be able to deploy pods. This is
because Kubernetes objects, like Deployments, use a Service Account
to schedule pods, instead of the user that created the Deployment.
user $ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/1 0 0 88s
user $ kubectl get replicasets
NAME DESIRED CURRENT READY AGE
nginx-cdcdd9f5c 1 0 0 92s
user $ kubectl describe replicasets nginx-cdcdd9f5c
...
Warning FailedCreate 48s (x15 over 2m10s) replicaset-controller Error creating: pods "nginx-cdcdd9f5c-" is forbidden: unable to validate against any pod security policy: []
For this deployment to be able to schedule pods, the service account defined wthin the deployment specification needs to be associated with a PSP policy. If a service account is not defined within a deployment spec, the default service account in a namespace is used.
This is the case in the deployment output above. As there is no service
account defined, a Rolebinding is needed to grant the default
service account in the default namespace to use the PSP policy.
Example RoleBinding to associate the unprivileged PSP policy
in MKE with the defaut service account in the default namespace:
admin $ cat <<EOF | kubectl create -f -
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: unprivileged-psp-role:defaultsa
namespace: default
roleRef:
kind: ClusterRole
name: unprivileged-psp-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: default
namespace: default
EOF
To allow the replica set to schedule pods within the cluster:
user $ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 6m11s
user $ kubectl get replicasets
NAME DESIRED CURRENT READY AGE
nginx-cdcdd9f5c 1 1 1 6m16s
user $ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-cdcdd9f5c-9kknc 1/1 Running 0 6m17s
user $ kubectl get pod nginx-cdcdd9f5c-9kknc -o json | jq -r '.metadata.annotations."kubernetes.io/psp"'
unprivileged
A common use case when using PSPs is to apply a particular policy to one
namespace, but not configure the rest. An example could be where an
admin might be want to configure keep the privileged policy for all
of the infrastructure namespaces but configure the unprivileged
policy for the end user namespaces. This can be done with the following
example:
In this demonstration cluster, infrastructure workloads are deployed in
the kube-system and the monitoring namespaces. End User
workloads are deployed in the default namespace.
admin $ kubectl get namespaces
NAME STATUS AGE
default Active 3d
kube-node-lease Active 3d
kube-public Active 3d
kube-system Active 3d
monitoring Active 3d
First, delete the ClusterRoleBinding that is applied by default in
MKE.
admin $ kubectl delete clusterrolebindings ucp:all:privileged-psp-role
Next, create a new ClusterRoleBinding that will enforce the
privileged PSP policy for all users and service accounts in the
kube-system and monitoring namespaces, where in this example
cluster the infrastructure workloads are deployed.
admin $ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ucp:infrastructure:privileged-psp-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-role
subjects:
- kind: Group
name: system:authenticated:kube-system
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:authenticated:monitoring
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:kube-system
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:monitoring
apiGroup: rbac.authorization.k8s.io
EOF
Finally, create a ClusterRoleBinding to allow all users who deploy
pods and deployments in the default namespace to use the
unprivileged policy.
admin $ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ucp:default:unprivileged-psp-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: unprivileged-psp-role
subjects:
- kind: Group
name: system:authenticated:default
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts:default
apiGroup: rbac.authorization.k8s.io
EOF
Now when the user deploys in the default namespace they will get the
unprivileged policy but when they deploy in the monitoring namespace
they will get the privileged policy.
user $ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: demopod
namespace: monitoring
spec:
containers:
- name: demopod
image: nginx
---
apiVersion: v1
kind: Pod
metadata:
name: demopod
namespace: default
spec:
containers:
- name: demopod
image: nginx
EOF
user $ kubectl get pods demopod -n monitoring -o json | jq -r '.metadata.annotations."kubernetes.io/psp"'
privileged
user $ kubectl get pods demopod -n default -o json | jq -r '.metadata.annotations."kubernetes.io/psp"'
unprivileged
To revert to the default MKE configuration, in which all MKE users and
service accounts use the privileged PSP, recreate the default
ClusterRoleBinding:
admin $ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ucp:all:privileged-psp-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-role
subjects:
- kind: Group
name: system:authenticated
apiGroup: rbac.authorization.k8s.io
- kind: Group
name: system:serviceaccounts
apiGroup: rbac.authorization.k8s.io
EOF
MKE admins or users with the correct permissions can create their own custom policies and attach them to MKE users or teams. This section highlights two potential use cases for custom PSPs. These two uses cases can be combined into the same policy. Note there are many more use cases with PSPs not covered in this document.
A common use case for PSPs is to prevent a user from deploying
containers that run with the root user. A PSP can be created to enforce
this with the parameter MustRunAsNonRoot.
admin $ cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: norootcontainers
spec:
allowPrivilegeEscalation: false
allowedHostPaths:
- pathPrefix: /dev/null
readOnly: true
fsGroup:
rule: RunAsAny
hostPorts:
- max: 65535
min: 0
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
EOF
If not done previously, the admin user must remove the
ClusterRoleBinding for the privileged policy, and then add a new
ClusterRole and RoleBinding to link a user to the new
norootcontainers policy.
# Delete the default privileged ClusterRoleBinding
admin $ kubectl delete clusterrolebindings ucp:all:privileged-psp-role
# Create a ClusterRole Granting Access to the Policy
admin $ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: norootcontainers-psp-role
rules:
- apiGroups:
- policy
resourceNames:
- norootcontainers
resources:
- podsecuritypolicies
verbs:
- use
EOF
# Define a User to attach to the No Root Policy
admin $ USER=jeff
# Create a RoleBinding attaching the User to the ClusterRole
admin $ cat <<EOF | kubectl create -f -
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: norootcontainers-psp-role:$USER
namespace: default
roleRef:
kind: ClusterRole
name: norootcontainers-psp-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: User
name: $USER
namespace: default
EOF
If a user tries to deploy a pod that runs as a root user, such as the
upstream nginx image, this should fail with a ConfigError.
user $ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: demopod
spec:
containers:
- name: demopod
image: nginx
EOF
user $ kubectl get pods
NAME READY STATUS RESTARTS AGE
demopod 0/1 CreateContainerConfigError 0 37s
user $ kubectl describe pods demopod
<..>
Error: container has runAsNonRoot and image will run as root
A second use case for PSPs is to prevent a user from deploying containers without a seccomp policy. By default, Kubernetes does not apply a seccomp policy to pods, so a default seccomp policy could be applied for all pods by a PSP.
admin $ cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: seccomppolicy
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default'
seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default'
spec:
allowPrivilegeEscalation: false
allowedHostPaths:
- pathPrefix: /dev/null
readOnly: true
fsGroup:
rule: RunAsAny
hostPorts:
- max: 65535
min: 0
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
EOF
If not done previously, the admin user must remove the
ClusterRoleBinding for the privileged policy, and then add a new
ClusterRole and RoleBinding to link a user to the new
applyseccompprofile policy.
# Delete the default privileged ClusterRoleBinding
admin $ kubectl delete clusterrolebindings ucp:all:privileged-psp-role
# Create a ClusterRole Granting Access to the Policy
admin $ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: applyseccompprofile-psp-role
rules:
- apiGroups:
- policy
resourceNames:
- seccomppolicy
resources:
- podsecuritypolicies
verbs:
- use
EOF
# Define a User to attach to the No Root Policy
admin $ USER=jeff
# Create a RoleBinding attaching the User to the ClusterRole
admin $ cat <<EOF | kubectl create -f -
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: applyseccompprofile-psp-role:$USER
namespace: default
roleRef:
kind: ClusterRole
name: applyseccompprofile-psp-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: User
name: $USER
namespace: default
EOF
As shown in the following example, if a user tries to deploy an
nginx pod without applying a seccomp policy as the pod metadata,
Kubernetes automatically applies a policy for the user.
user $ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: demopod
spec:
containers:
- name: demopod
image: nginx
EOF
user $ kubectl get pods
NAME READY STATUS RESTARTS AGE
demopod 1/1 Running 0 16s
user $ kubectl get pods demopod -o json | jq '.metadata.annotations."seccomp.security.alpha.kubernetes.io/pod"'
"docker/default"