Perform the CA rotation#

The following procedure rotates all four CAs at the same time. Rotating all four requires no more downtime than rotating the Kubernetes CA alone.

Before you begin, complete every task in Prepare for CA rotation.

To rotate a subset of the CAs:

In step 2, run only the file deletions for the CAs you are rotating.
If you are rotating only the etcd CA, the front-proxy CA, or both, skip steps 3, 8, and 9. These steps re-establish worker node identity only.

All other steps apply to every rotation.

Warning

On a single-node cluster, reboot the machine wherever this procedure instructs you to start or restart k0s on a controller node. A single-node MKE 4 cluster runs its controller with the --enable-worker flag, and thus the node is both a controller node and a worker node. Restarting k0s is insufficient, as the embedded kubelet does not load the new identity. The same requirement applies to any controller node that you configured to run as a worker.

Step 1: Stop k0s on all nodes#

Stop the worker nodes before the controller nodes, so that the worker nodes do not log a large volume of failures against an unavailable API server.

On each Linux worker node, stop k0s:
```
sudo k0s stop
```
On each Windows worker node, stop the k0s worker service. Do not use the k0s stop command on Windows:
```
Stop-Service k0sworker -Force
```
Note

This is the first PowerShell command in the procedure. If the output reports that Stop-Service is not recognized, the session is still in cmd.exe. Enter PowerShell, as described in Identify the Windows worker paths.
On each controller node, stop k0s:
```
sudo k0s stop
```
On each Linux node, verify that no k0s service is running:
```
systemctl is-active k0sworker k0scontroller
```
The output shows inactive for both services, or unknown for a service that is not installed on the node.
On each Windows worker node, verify that the k0s worker service is stopped:
```
Get-Service k0sworker
```
The Status field shows Stopped.

Step 2: Delete the CA material on all controller nodes#

Warning

Delete only the files listed in this step. Do not delete the pki directory. The $DATA_DIR/pki directory also contains MKE-owned material that k0s does not regenerate:

client-ca.{crt,key}, the trust anchor for your users
custom-ca.crt and custom-apiserver-tls.{crt,key}, if you supply your own API serving certificate

Deleting the pki directory destroys these files and breaks user authentication. The only recovery method is to restore from backup.

On each controller node, verify that the DATA_DIR variable resolves on the node:
```
ls -l "$DATA_DIR/pki/ca.crt"
```
The output lists the file. If the output reports No such file or directory, the DATA_DIR variable is incorrect or unset in the current shell. Set the variable, as described in Set the path variables on each node, and run the check again. Do not continue until the check lists the file, as the deletion commands otherwise remove nothing and report no error.

On each controller node, delete the CA material:

# Kubernetes CA
sudo rm -f "$DATA_DIR/pki/ca.crt" "$DATA_DIR/pki/ca.key"

# ServiceAccount key pair - BOTH files, see warning below
sudo rm -f "$DATA_DIR/pki/sa.key" "$DATA_DIR/pki/sa.pub"

# front-proxy CA
sudo rm -f "$DATA_DIR/pki/front-proxy-ca.crt" "$DATA_DIR/pki/front-proxy-ca.key"

# etcd CA and the certificates it signs
sudo rm -f "$DATA_DIR/pki/etcd/ca.crt" "$DATA_DIR/pki/etcd/ca.key"
sudo rm -f "$DATA_DIR/pki/etcd/server.crt" "$DATA_DIR/pki/etcd/server.key"
sudo rm -f "$DATA_DIR/pki/etcd/peer.crt" "$DATA_DIR/pki/etcd/peer.key"
sudo rm -f "$DATA_DIR/pki/apiserver-etcd-client.crt" "$DATA_DIR/pki/apiserver-etcd-client.key"

Warning

Delete both files of the ServiceAccount key pair. k0s regenerates the key pair only when both sa.key and sa.pub are absent. If you delete only one file, k0s retains the existing key pair, and the procedure completes without error but does not rotate the key pair.

Do not delete the $DATA_DIR/pki/etcd/jwt.key and jwt.pub files. This key pair signs etcd join tokens and is unrelated to the etcd CA.

You do not need to delete the leaf certificates that the Kubernetes CA signs: admin, scheduler, ccm, apiserver-kubelet-client, front-proxy-client, konnectivity, server, and k0s-api. On start, k0s detects that their issuer is missing and reissues them.

Step 3: Delete the kubelet identity on all worker nodes#

Skip this step if you are rotating only the etcd CA, the front-proxy CA, or both.

On each Linux worker node, and on each controller node that runs with the --enable-worker flag:

Verify that both paths resolve on the node:
```
sudo ls -ld "$DATA_DIR/kubelet.conf" "$KUBELET_PKI_DIR"
```
The output lists both paths. If either path reports No such file or directory before you delete anything, the corresponding variable is incorrect or unset in the current shell. Correct the variable before you continue. The same message indicates success after the deletion, and thus an incorrect path produces a verification that passes without anything being deleted.

Delete the kubelet identity:

sudo rm -f "$DATA_DIR/kubelet.conf"
sudo rm -rf "$KUBELET_PKI_DIR"

Verify that both paths are deleted:
```
sudo ls -ld "$DATA_DIR/kubelet.conf" "$KUBELET_PKI_DIR" 2>&1
```
The output reports No such file or directory for both paths.

On each Windows worker node, confirm that the kubelet identity exists, delete it, and confirm the deletion, using the fixed Windows paths:

Get-Item C:\var\lib\k0s\kubelet.conf, C:\var\lib\k0s\kubelet\pki

Remove-Item -Force C:\var\lib\k0s\kubelet.conf
Remove-Item -Recurse -Force C:\var\lib\k0s\kubelet\pki

Get-Item C:\var\lib\k0s\kubelet.conf, C:\var\lib\k0s\kubelet\pki -ErrorAction SilentlyContinue

The first command lists both items, and the last command returns nothing. If the kubelet PKI path you confirmed earlier is C:\var\lib\kubelet\pki, substitute that path.

Step 4: Start the first controller node#

Select one controller node to generate the new CAs.

On the selected controller node, start k0s:
```
sudo k0s start
```
If the controller node runs with the --enable-worker flag, including on a single-node cluster, reboot the machine instead:
```
sudo reboot
```
Important

A reboot ends the shell session, and thus the DATA_DIR and KUBELET_PKI_DIR variables are unset when you reconnect. Set the variables again before you run any other command on the node, as described in Set the path variables on each node. All remaining steps on this controller node use them.
Wait for the API server to respond. The k0s kubectl command reads the admin kubeconfig from the k0s data directory, and thus requires no path:
```
sudo k0s kubectl get --raw='/readyz?verbose'
```
The check list ends with readyz check passed. The etcd ok and etcd-readiness ok lines confirm that the etcd CA material regenerated correctly.

Step 5: Verify that k0s generated the new CAs#

Do not continue until this verification passes. If the CAs did not regenerate, all subsequent steps distribute outdated material.

Verify the certificate dates:
```
for f in ca.crt front-proxy-ca.crt etcd/ca.crt; do
  echo "== $f"
  sudo openssl x509 -in "$DATA_DIR/pki/$f" -noout -subject -issuer -dates
done
```
The notBefore value for all three certificates falls within the last few minutes. If any certificate shows an earlier date, that CA did not rotate. Investigate the cause before you proceed.
Verify that the ServiceAccount key pair is new:
```
sudo stat -c '%n %y' "$DATA_DIR/pki/sa.key" "$DATA_DIR/pki/sa.pub"
```
Both timestamps match the time of the k0s start. If either timestamp is earlier, sa.pub was likely left in place. Stop k0s, return to step 2, delete both files, and start k0s again.

Step 6: Distribute the new CAs to the remaining controller nodes#

Warning

Copy only the CA files listed in this step. Do not copy the etcd peer certificate. The pki/etcd/peer.crt certificate contains the peer address of the node as both its subject and its only SAN. A copied peer certificate does not match the node that presents it, which causes etcd peer connections to fail with errors that are difficult to diagnose. Each controller node must generate its own peer certificate.

Copy the following files from the first controller node to each remaining controller node:

$DATA_DIR/pki/ca.crt
$DATA_DIR/pki/ca.key
$DATA_DIR/pki/sa.key
$DATA_DIR/pki/sa.pub
$DATA_DIR/pki/front-proxy-ca.crt
$DATA_DIR/pki/front-proxy-ca.key
$DATA_DIR/pki/etcd/ca.crt
$DATA_DIR/pki/etcd/ca.key

On each remaining controller node, set the file ownership and permissions. The CA keys must not be world-readable:

sudo chmod 0600 "$DATA_DIR/pki/ca.key" "$DATA_DIR/pki/sa.key" \
                "$DATA_DIR/pki/front-proxy-ca.key" "$DATA_DIR/pki/etcd/ca.key"
sudo chmod 0644 "$DATA_DIR/pki/ca.crt" "$DATA_DIR/pki/sa.pub" \
                "$DATA_DIR/pki/front-proxy-ca.crt" "$DATA_DIR/pki/etcd/ca.crt"

Start k0s on each remaining controller node, one node at a time. Wait for each node to become ready before you start the next:
```
sudo k0s start
```
If the controller node runs with the --enable-worker flag, reboot the machine instead.

Step 7: Verify the control plane and etcd quorum#

Verify that every controller node is an etcd member:
```
sudo k0s etcd member-list --data-dir="$DATA_DIR"
```
The output lists every controller node.
On any controller node, verify the node status:
```
sudo k0s kubectl get nodes
```
All controller nodes show Ready. Worker nodes show NotReady or are absent until they rejoin the cluster.

Step 8: Rejoin the worker nodes#

Skip this step if you are rotating only the etcd CA, the front-proxy CA, or both.

On a controller node, create a worker bootstrap token in two forms. Linux worker nodes use the decoded kubeconfig, and Windows worker nodes use the token exactly as the k0s token create command prints it:
```
sudo touch /tmp/rejoin-token /tmp/rejoin-token.raw
sudo chmod 0600 /tmp/rejoin-token /tmp/rejoin-token.raw

sudo k0s token create --expiry 1h --role worker | sudo tee /tmp/rejoin-token.raw >/dev/null
sudo cat /tmp/rejoin-token.raw | base64 -d | gunzip | sudo tee /tmp/rejoin-token >/dev/null
```
The token must remain valid until the last worker node first contacts the API server. An expiry of one hour suits a small number of worker nodes. For a large cluster that you reboot in batches, set the --expiry flag to cover the entire rejoin window. If the token expires during the rejoin, create a new token and continue with the remaining worker nodes.
On each Linux worker node, rejoin the node:
1. Copy /tmp/rejoin-token to the worker node as $DATA_DIR/kubelet-bootstrap.conf.
2. Set the file permissions and ownership, and reboot the node:
```
sudo chmod 0600 "$DATA_DIR/kubelet-bootstrap.conf"
sudo chown root:root "$DATA_DIR/kubelet-bootstrap.conf"
sudo reboot
```
On each Windows worker node, rejoin the node:
1. Copy /tmp/rejoin-token.raw to the worker node as C:\etc\k0s\k0stoken. The k0s worker service reads its join token from this path.
2. Restrict the file ACL to SYSTEM and Administrators, as mkectl does on every apply, and restart the node:
```
icacls C:\etc\k0s\k0stoken /inheritance:r /grant '*S-1-5-18:(F)' /grant '*S-1-5-32-544:(F)'

Restart-Computer -Force
```
Monitor the worker nodes as they rejoin the cluster:
```
sudo k0s kubectl get nodes -w
```

Note

The reissued kubelet certificate on a worker node carries a notBefore value approximately five minutes earlier than the time that the node rejoined. The Kubernetes signer backdates certificates to tolerate clock skew, and thus the earlier value does not indicate that an outdated certificate remains. To confirm that a worker node uses the new CA, verify that its certificate chains to the new CA:

openssl verify -CAfile ca.crt kubelet-client-current.pem

The output shows OK.

Step 9: Clean up the bootstrap tokens#

Skip this step if you skipped step 8.

After every worker node shows Ready, remove the bootstrap files and invalidate the token to prevent its reuse.

On each Linux worker node, delete the bootstrap file:
```
sudo rm -f "$DATA_DIR/kubelet-bootstrap.conf"
```
Do not delete C:\etc\k0s\k0stoken from Windows worker nodes. The k0s worker service reads its join token from this path, and mkectl secures the file on every apply rather than removing it. The restricted ACL, together with the token invalidation, renders the file unusable.

On a controller node, invalidate the token:

sudo k0s token list --role worker
sudo k0s token invalidate <token-id>

Delete /tmp/rejoin-token and /tmp/rejoin-token.raw from the controller node and from any location where you staged either file in transit. Both files are cluster join credentials.

After you complete the rotation, verify the result. For more information, refer to Verify the CA rotation.