Runbook: MKE 3.9.5 to MKE 4.2.0 upgrade, etcd v2store-clean workaround#

Warning

This issue is fixed in MKE 4.2.1. The workaround for the issue addressed herein only applies if you must upgrade with mkectl 4.2.0.

Upgrades from MKE 3.9.5 with mkectl 4.2.0 fail at the [Migrate MKE 3 Nodes] point. The bundled etcdutl offers No custom content found in both v2store and WAL records., however mkectl only accepts No custom content found in v2store, so it exhausts its retry budget and rolls back automatically.

MKE 3.9.5 ships with etcd 3.5.32, which rewords the etcdutl output. The v2 store is genuinely clean and the migration is safe to run; mkectl 4.2.0, though, does not recognize the newer wording. The upgrade fails safe, and rolls back to MKE 3.9.5 with no data loss.

Note

Only MKE 3.9.5 is affected by this issue, and all other MKE 3.9.x releases must not use this workaround. Note that clusters that are running MKE 3.8.2 must first be upgraded to MKE 3.9.x.

Mechanism#

On each manager, the running ucp-kv container (etcd 3.5.32) is replaced with the MKE 3.9.2 image (etcd 3.5.21) by retagging over the configured tag and letting ucp-agent recreate the container. Recreating on 3.5.21 rewrites the on-disk etcd state so that the v2store check for the migration reads clean in a form mkectl 4.2.0 accepts, and it passes in one iteration.

Before you start#

Complete the following tasks before you run the etcd downgrade script:

Back up etcd.
Verify that all manager nodes are healthy etcd members and that the etcd cluster has quorum.
Verify that the node can authenticate to the registry that serves ucp-etcd. The script does not run docker login.
Airgapped environments only: Stage the ucp-etcd:3.9.2 image in the same repository as ucp-etcd:3.9.5. The 3.9.5 bundle does not include the 3.9.2 image. The script detects the repository by running docker inspect ucp-kv, so you do not specify it. The script does not push images to the registry.

The script downgrades the etcd server, not only etcdutl

The ucp-etcd:3.9.2 image replaces the etcd server with etcd 3.5.21. The 3.5.21 v2store checker inspects v2 WAL data less thoroughly than the 3.5.32 checker. Verify that this is acceptable for your cluster before you proceed. The downgrade is temporary: after the upgrade completes, MKE 4 runs the etcd version that k0s manages.

Apply the workaround#

Apply to one manager at a time. Move to the next only after the current manager is healthy and quorum is confirmed. Recreating ucp-kv briefly interrupts that manager's etcd member.
Only one network pull occurs, for the 3.9.2 image. ucp-agent recreates ucp-kv with a missing-only pull, so the local retag holds, and recreate and rollback both use local images.
The script pulls and verifies the replacement image first. If 3.9.2 cannot be reached, it aborts with the cluster completely unchanged.
Apply to all managers, then start the mkectl upgrade promptly.

Step 1. Run the etcd v2store mitigation script on each manager#

Run the script on one manager at a time. Complete and verify each manager before you move to the next.

Download ucp-etcd-v2store-mitigation.sh and copy it to each manager.
On the first manager, make the script executable and run it in apply mode:
```
chmod +x ucp-etcd-v2store-mitigation.sh
sudo ./ucp-etcd-v2store-mitigation.sh apply
```
The script prompts for confirmation at four points:
- Before it removes the ucp-kv container
- After it recreates the container
- After the v2store check
- Before you move to the next manager
To skip the prompts, pass -y or --assume-yes.
Confirm that the script output ends with etcdutl version: 3.5.21 and Manager complete.

The script also prints a v2store check result. This result varies with cluster state and is informational only. It is not a pass condition.
Verify that ucp-kv is healthy and that etcd has quorum.
Repeat steps 2 through 4 on each remaining manager.

Other modes

The etcd v2store mitigation script provides two other modes:

Mode	When to use it
`rollback`	The upgrade is delayed or has failed and you need the original image back. Run it one manager at a time.
`cleanup`	Only if you rolled back and do not intend to retry. It removes the rollback tag and state file.

To retry after a rollback, run apply again in full. Do not run cleanup first. Each apply re-points the tag from scratch, and it refuses to run if ucp-kv already reports the replacement etcd version.
You do not need to run cleanup after a successful upgrade. Once the upgrade completes, etcd runs under k0s and containerd, the MKE 3 Docker layer is gone, and the tags and state file the script created are inert.

Step 2. Upgrade the cluster to MKE 4#

Start the upgrade only after you complete Step 1 on every manager.

Before you begin the upgrade

Run mkectl from a host that can reach the internal registry and can SSH to every node. The pre-checks fetch some artifact manifests from the mkectl host, even when you pass --skip-image-presence-check to skip the bulk image presence check.
For airgapped clusters, configure the internal registry to present a leaf TLS certificate signed by a separate CA. The upgrade rejects a self-signed certificate that acts as its own CA.
If MKE 3 ingress is disabled, add the following flags to the upgrade command:
```
--gateway-http-node-port 33000 --gateway-https-node-port 33001
```

Run mkectl upgrade with the flags for your environment.

For an airgapped cluster:

mkectl upgrade --hosts-path hosts.yaml \
  --mke3-admin-username admin --mke3-admin-password '<pw>' \
  --mke3-airgapped=true \
  --image-registry <registry>/mke \
  --chart-registry oci://<registry>/mke \
  --image-registry-ca-file <ca.crt> \
  --chart-registry-ca-file <ca.crt> \
  --external-address <cluster-endpoint>

For a cluster that is not airgapped:

mkectl upgrade --hosts-path hosts.yaml \
  --mke3-admin-username admin --mke3-admin-password '<pw>' \
  --external-address <cluster-endpoint>

At the [Migrate MKE 3 Nodes] step, confirm that the log reports:
```
v2store clean on all managers after 1 iteration(s)
```
Confirm that the upgrade completes all 17 steps.