Force certificate renewal for Kubernetes components

Applies to k0s-based clusters only

k0s reissues the certificates of the Kubernetes control plane components, such as the Kubernetes API server serving certificate and the etcd certificates, every time the k0scontroller systemd service restarts. Since MOSK restarts the service on every configuration change of the Kubernetes underlay, these certificates are normally renewed without operator intervention.

Use the procedure below when the certificates must be reissued but no configuration change is planned. For example, when the certificates are about to expire or when the Kubernetes API endpoint host name has changed and the serving certificate still carries the previous name in its subject alternative name (SAN) list. As a result, clients that do not use Server Name Indication (SNI) fail host name verification. A typical symptom is the MOSK management console failing with the HTTP 502 error and the upstream SSL certificate does not match message in its logs.

Caution

The procedure covers certificate renewal of the Kubernetes components only. It does not affect custom certificates configured for cluster applications, including custom k0s certificates served through SNI. For reference, see Configure TLS certificates for cluster applications and Workflow of custom k0s certificates configuration.

Warning

The procedure restarts all Kubernetes control plane components on each control plane node. You must process the nodes strictly one by one and only with maintenance mode enabled, so that the cluster API remains available and the etcd quorum is preserved.

Until the last node is processed, the nodes serve different versions of the serving certificate, so clients may observe transient certificate verification failures.

Plan the procedure as a maintenance operation and verify that all cluster nodes are Ready before proceeding to the steps below.

To force certificate renewal for Kubernetes components:

Note

k0s reissues the certificates using the same CA, so the cluster trust chain and the existing kubeconfig files remain valid.

Log in to a node that contains kubeconfig of the required management cluster and obtain the control plane machines of the cluster:
```
kubectl get machines -l cluster.sigs.k8s.io/control-plane -o wide
```
Enable maintenance mode on the cluster as described in Enable cluster and machine maintenance mode.
Repeat the following steps for every control plane machine, one machine at a time:
1. Enable maintenance mode on the machine as described in Enable cluster and machine maintenance mode and wait until its node is cordoned and drained.
2. Log in to the node of the machine through SSH.
3. Capture the current parameters of the Kubernetes API server serving certificate:
```
sudo openssl x509 -in /var/lib/k0s/pki/server.crt -noout -serial -dates -ext subjectAltName
```
4. Identify the certificate authority (CA) of the k0s public key infrastructure (PKI):
```
sudo openssl x509 -in /var/lib/k0s/pki/ca.crt -noout -subject
```
5. If the output does not contain CN = kubernetes-ca, the PKI was inherited from Mirantis Kubernetes Engine (MKE) during cluster migration to k0s. k0s does not reissue certificates signed by a CA created before migration. Therefore, remove the affected certificates manually:
```
for cert in admin apiserver-kubelet-client apiserver-etcd-client \
            ccm konnectivity scheduler server k0s-api \
            front-proxy-client etcd/server etcd/peer; do
    sudo rm -f "/var/lib/k0s/pki/${cert}.crt" "/var/lib/k0s/pki/${cert}.key"
done
```
  Warning
  
  Remove only the certificates listed above. Never remove the CA certificates and keys (ca, etcd/ca, front-proxy-ca, client-ca), or the sa and etcd/jwt key pairs.
  
  Removing a CA breaks the cluster trust, and removing the service account keys invalidates all issued service account tokens.
6. Restart the k0scontroller service:
```
sudo systemctl restart k0scontroller
```
7. Verify that the certificates are reissued:
```
sudo openssl x509 -in /var/lib/k0s/pki/server.crt -noout -serial -dates -ext subjectAltName
```
  - The serial number must differ from the one captured before the restart
  - The validity period must start at the time of the restart
  - The SAN list must contain the expected host names and IP addresses of the cluster API endpoint
8. Disable maintenance mode on the machine as described in Enable cluster and machine maintenance mode and wait until the node becomes Ready before proceeding to the next machine.
Disable maintenance mode on the cluster as described in Enable cluster and machine maintenance mode.
Back up k0s as described in Configure backups of the Kubernetes underlay.