Kubernetes network encryption

MKE provides data-plane level IPSec network encryption to securely encrypt application traffic in a Kubernetes cluster. This secures application traffic within a cluster when running in untrusted infrastructure or environments. It is an optional feature of MKE that is enabled by deploying the SecureOverlay components on Kubernetes when using the default Calico driver for networking with the default IPIP tunneling configuration.

Kubernetes network encryption is enabled by two components in MKE:

• SecureOverlay Agent

• SecureOverlay Master

The SecureOverlay Agent is deployed as a per-node service that manages the encryption state of the data plane. The Agent controls the IPSec encryption on Calico IPIP tunnel traffic between different nodes in the Kubernetes cluster. The Master is deployed on an MKE manager node and acts as the key management process that configures and periodically rotates the encryption keys.

Kubernetes network encryption uses AES Galois Counter Mode (AES-GCM) with 128-bit keys by default.

You must deploy the SecureOverlay Agent and Master on MKE to enable encryption, as it is not enabled by default. You can enable or disable encryption at any time during the cluster lifecycle. However, be aware that enabling or disabling encryption can cause temporary traffic outages between Pods, lasting up to a few minutes. When enabled, Kubernetes Pod traffic between hosts is encrypted at the IPIP tunnel interface in the MKE host.

Kubernetes network encryption is supported on the following platforms:

Platform	Encryption support
MKE 3.1 and later	Yes
Kubernetes 1.11 and later	Yes
On-premises	Yes
AWS	Yes
GCE	Yes
All MKE-supported Linux OSes	Yes
Azure	No
Unmanaged CNI plugins	No

Configure maximum transmission units

Maximum transmission units (MTUs) are the largest packet length that a container will allow. Before deploying the SecureOverlay components, verify that Calico is configured so that the IPIP tunnel MTU leaves sufficient room for the encryption overhead. Encryption adds 26 bytes of overhead, but every IPSec packet size must be a multiple of 4 bytes. IPIP tunnels require 20 bytes of encapsulation overhead. The IPIP tunnel interface MTU must be no more than EXTMTU - 46 - ((EXTMTU - 46) modulo 4), where EXTMTU is the minimum MTU of the external interfaces. An IPIP MTU of 1452 should generally be safe for most deployments.

In the MKE configuration file, update the ipip_mtu parameter with the new MTU:

[cluster_config]
 ...
 ipip_mtu = "1452"
 ...

Configure SecureOverlay

Once the cluster node MTUs are properly configured, deploy the SecureOverlay components to MKE using either the MKE configuration file or the SecureOverlay YAML file.

To configure SecureOverlay using the MKE configuration file:

Set the value of secure_overlay in the MKE configuration file cluster_config table to true.

To configure SecureOverlay using the SecureOverlay YAML file:

Run the following procedure at the time of cluster installation, prior to starting any workloads.

1. Copy the contents of the SecureOverlay YAML file into a YAML file called ucp-secureoverlay.yaml.

   SecureOverlay YAML

   # Cluster role for key management jobs
   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: ucp-secureoverlay-mgr
   rules:
     - apiGroups: [""]
       resources:
         - secrets
       verbs:
         - get
         - update
   ---
   # Cluster role binding for key management jobs
   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: ucp-secureoverlay-mgr
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: ucp-secureoverlay-mgr
   subjects:
   - kind: ServiceAccount
     name: ucp-secureoverlay-mgr
     namespace: kube-system
   ---
   # Service account for key management jobs
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: ucp-secureoverlay-mgr
     namespace: kube-system
   ---
   # Cluster role for secure overlay per-node agent
   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: ucp-secureoverlay-agent
   rules:
     - apiGroups: [""]
       resources:
         - nodes
       verbs:
         - get
         - list
         - watch
   ---
   # Cluster role binding for secure overlay per-node agent
   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: ucp-secureoverlay-agent
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: ucp-secureoverlay-agent
   subjects:
   - kind: ServiceAccount
     name: ucp-secureoverlay-agent
     namespace: kube-system
   ---
   # Service account secure overlay per-node agent
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: ucp-secureoverlay-agent
     namespace: kube-system
   ---
   # K8s secret of current key configuration
   apiVersion: v1
   kind: Secret
   metadata:
     name: ucp-secureoverlay
     namespace: kube-system
   type: Opaque
   data:
     keys: ""
   ---
   # DaemonSet for secure overlay per-node agent
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: ucp-secureoverlay-agent
     namespace: kube-system
     labels:
       k8s-app: ucp-secureoverlay-agent
   spec:
     selector:
       matchLabels:
         k8s-app: ucp-secureoverlay-agent
     updateStrategy:
       type: RollingUpdate
     template:
       metadata:
         labels:
           k8s-app: ucp-secureoverlay-agent
         annotations:
           scheduler.alpha.kubernetes.io/critical-pod: ''
       spec:
         hostNetwork: true
         priorityClassName: system-node-critical
         terminationGracePeriodSeconds: 10
         serviceAccountName: ucp-secureoverlay-agent
         containers:
         - name: ucp-secureoverlay-agent
           image: docker/ucp-secureoverlay-agent:3.1.0
           securityContext:
             capabilities:
               add: ["NET_ADMIN"]
           env:
           - name: MY_NODE_NAME
             valueFrom:
               fieldRef:
                 fieldPath: spec.nodeName
           volumeMounts:
           - name: ucp-secureoverlay
             mountPath: /etc/secureoverlay/
             readOnly: true
         volumes:
         - name: ucp-secureoverlay
           secret:
             secretName: ucp-secureoverlay
   ---
   # Deployment for manager of the whole cluster (primarily to rotate keys)
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ucp-secureoverlay-mgr
     namespace: kube-system
   spec:
     selector:
       matchLabels:
         app: ucp-secureoverlay-mgr
     replicas: 1
     template:
       metadata:
         name: ucp-secureoverlay-mgr
         namespace: kube-system
         labels:
           app: ucp-secureoverlay-mgr
       spec:
         serviceAccountName: ucp-secureoverlay-mgr
         restartPolicy: Always
         containers:
         - name: ucp-secureoverlay-mgr
           image: docker/ucp-secureoverlay-mgr:3.1.0
   

2. Download and configure the client bundle.

3. Enable network encryption:

   kubectl apply -f ucp-secureoverlay.yml
   

Note

To remove network encryption from the system, issue the following command:

kubectl delete -f ucp-secureoverlay.yml