Skip to content

Prepare for CA rotation#

Complete the following preparation before you perform the CA rotation.

Derive the cluster paths#

Every path in the CA rotation procedure depends on the directories configured for your cluster. Do not use /var/lib/k0s as a literal value, as the data directory is configurable. If you use an incorrect path, the deletion commands remove nothing and report success.

On the workstation where you run mkectl, derive the data directory and the kubelet root directory:

mkectl config get > mke4-current.yaml

# Data directory: spec.k8sProviderSpecs.dataDir, default /var/lib/k0s
DATA_DIR=$(yq -r '.spec.k8sProviderSpecs.dataDir // "/var/lib/k0s"' mke4-current.yaml)

# Kubelet root directory: spec.kubelet.kubeletRootDir
KUBELET_DIR=$(yq -r '.spec.kubelet.kubeletRootDir // ""' mke4-current.yaml)

echo "data dir:     $DATA_DIR"
echo "kubelet root: ${KUBELET_DIR:-<unset - see below>}"

Important

Confirm the location of the kubelet PKI before you begin. If spec.kubelet.kubeletRootDir is unset, the kubelet PKI is at either <dataDir>/kubelet/pki or /var/lib/kubelet/pki, depending on the history of the cluster. On one Linux worker node, check both locations and use the one that exists:

sudo ls -d "$DATA_DIR"/kubelet/pki /var/lib/kubelet/pki 2>/dev/null

On Windows worker nodes, check both of the following locations and use the one that exists. The second location is a legacy path that some earlier resets leave behind:

Get-Item C:\var\lib\k0s\kubelet\pki, C:\var\lib\kubelet\pki -ErrorAction SilentlyContinue

Record the kubelet PKI directory alongside the data directory:

  • If kubeletRootDir is set, the kubelet PKI directory is that path with /pki appended.
  • If kubeletRootDir is unset, the kubelet PKI directory is the path that the preceding check found.

Edit the following value to match the path you confirmed:

KUBELET_PKI_DIR=/var/lib/kubelet/pki

echo "data dir:     $DATA_DIR"
echo "kubelet pki:  $KUBELET_PKI_DIR"

Identify the Windows worker paths#

Important

The spec.k8sProviderSpecs.dataDir setting does not apply to Windows worker nodes. MKE 4 uses a fixed data directory on Windows, regardless of the configuration, and thus the DATA_DIR value you derived applies to Linux nodes only. Use the following literal paths on every Windows worker node:

Item Path on Windows
Data directory C:\var\lib\k0s
Kubelet identity C:\var\lib\k0s\kubelet.conf
Kubelet PKI C:\var\lib\k0s\kubelet\pki (legacy: C:\var\lib\kubelet\pki)
Join token C:\etc\k0s\k0stoken
k0s binary C:\Program Files\k0s\k0s.exe

Important

All Windows commands in the CA rotation procedure are PowerShell commands. An SSH session on a Windows node opens in cmd.exe, where these commands fail with the error is not recognized as an internal or external command. Before you run any command in the procedure, enter PowerShell on every Windows worker node:

powershell

The prompt changes to PS C:\...>. Enter PowerShell again in any session that you reconnect following a reboot. To run a single command without entering PowerShell, wrap the command instead:

powershell.exe -NonInteractive -NoProfile -ExecutionPolicy Unrestricted -Command "<command>"

Set the path variables on each node#

Warning

The DATA_DIR and KUBELET_PKI_DIR variables exist only in the shell that set them. Every deletion and check in the CA rotation procedure runs in a separate shell on a controller node or worker node, where these variables are unset. An unset variable expands to an empty string. For example, rm -f "$DATA_DIR/pki/ca.crt" targets /pki/ca.crt, removes nothing, and exits with status zero. The rotation then appears to succeed without rotating anything.

To set the path variables on each node:

  1. On the workstation, print the two variable assignments:

    printf 'export DATA_DIR=%q\nexport KUBELET_PKI_DIR=%q\n' "$DATA_DIR" "$KUBELET_PKI_DIR"

  2. Paste both lines into the root shell on each controller node and each Linux worker node before you run any command in the procedure.

  3. Paste both lines again in any session that you reconnect following a reboot.

Each step that uses these variables begins with a check that confirms they resolve to real paths on the node.

Windows worker nodes do not require the path variables. The Windows commands in the procedure use the fixed paths listed in Identify the Windows worker paths.

Verify cluster health#

Rotating the CAs on a degraded cluster can result in the loss of the cluster. A node that is unreachable before the rotation cannot rejoin the cluster afterward.

Verify the health of the cluster:

mkectl status
mkectl check hosts

Warning

Do not proceed if either command reports a problem. Every controller node and every worker node must be reachable and healthy. Repair any unhealthy node, or remove it from the cluster, before you rotate the CAs. If an unhealthy node fails to rejoin, its workloads cannot be rescheduled, and you cannot readmit the node without its certificates.

Create a backup#

The backup is your only rollback method. Create the backup immediately before you begin the rotation, so that it includes every configuration and workload change.

To create the backup, refer to Backup. MKE 4 writes the backup archive to the machine that runs mkectl, rather than to the cluster nodes. Before you continue, store a second copy of the archive on a different machine.

Important

Two backup limitations apply directly to CA rotation:

  • You can restore a backup only onto the same MKE 4 version that created it. A backup taken before an upgrade cannot serve as a rollback for a rotation performed after that upgrade.
  • MKE 4 may not be able to back up a cluster that has crashed. For this reason, verify cluster health before you create the backup.

Record the current certificate state#

Record the current state of the cluster certificates:

mkectl certs status --include-node-pki -o json > certs-before.json

Note

The --include-node-pki flag requires SSH access to the control plane hosts. Without SSH access, the command does not inspect the Kubernetes, etcd, or kubelet certificates, and the report appears identical to that of a healthy cluster. Verify that the output states that the node tier was inspected.

Retain the certs-before.json file. After the rotation, you compare it against the post-rotation state to confirm that the rotation took effect. For more information, refer to Verify the CA rotation.