Skip to content

Perform the MKE 3 to MKE 4 Upgrade#

Note

Starting with MKE 4.2.0, only MKE 3.9.x clusters can be upgraded to MKE 4.2.0 and later. For more information, refer to Upgrade from MKE 3.7, 3.8, or 3.9.

An upgrade from MKE 3 to MKE 4 uses the MKE 4 CLI (mkectl) to perform the following tasks:

  • Run pre-upgrade checks to verify the upgradability of the cluster.
  • Carry out pre-upgrade migrations to prepare the cluster for an upgrade from a hyperkube-based MKE 3 cluster to a k0s-based MKE 4 cluster.
  • Upgrade manager nodes to k0s.
  • Upgrade worker nodes to k0s.
  • Carry out post-upgrade cleanup to remove MKE 3 components.
  • Output the new mke4.yaml configuration file.

As MKE 3 clusters do not store MKE 4 pull credentials, when you upgrade against a registry that requires authentication you must supply the pull credentials at upgrade time, using the same flags and environment variables as with the mkectl apply command:

mkectl upgrade \
  --hosts-path <path-to-hosts.yaml> \
  --mke3-admin-username <admin> --mke3-admin-password <password> \
  --external-address <load-balancer-address> \
  --image-registry <registry-host>/mke \
  --chart-registry oci://<registry-host>/mke \
  --image-registry-username <username> --image-registry-password <password> \
  --chart-registry-username <username> --chart-registry-password <password>

Password Protection

To assure that passwords are not included in your shell history and the process argument list, export them rather than passing --*-registry-password:

export MKE_IMAGE_REGISTRY_PASSWORD=<password>
export MKE_CHART_REGISTRY_PASSWORD=<password>
mkectl upgrade \
  --hosts-path <path-to-hosts.yaml> \
  --mke3-admin-username <admin> --mke3-admin-password <password> \
  --external-address <load-balancer-address> \
  --image-registry <registry-host>/mke \
  --chart-registry oci://<registry-host>/mke \
  --image-registry-username <username> \
  --chart-registry-username <username>

The mkectl CLI writes the resolved credentials into the containerd registry configuration on every migrated node and stores them in the credentialsSecretName Secret in the mke namespace. For MKE 3 to MKE 4 upgrades, MKE chooses the mke-image-registry-credentials and mke-chart-registry-credentials secret names, respectively, which ensures that k0s binary, image, and chart pulls authenticate both during and after the migration. The same rules apply as for a fresh install: the registry must be served over HTTPS, you must supply each username and password together, and image and chart credentials must match when registries share a host.

Note

As MKE 4 only requires an account with read permissions, Mirantis strongly recommends that the credentials you provide correspond to a read-only account.

Specific to Windows worker nodes

  • Important: MKE 4 does not support custom root directories for containerd and kubelet on Windows, and as a result migrated Windows nodes are set up to use MKE 4 default root directories. This requires that the images be re-pulled, which may extend the downtime for any Windows workloads. To minimize disruption, make sure that all Windows workload images are available in a registry to which the cluster has access.
  • Any Windows worker nodes in the source MKE 3 cluster are upgraded to k0s at the same time the Linux worker nodes are upgraded.
  • Windows nodes reboot as part of the upgrade process, and thus any Windows workloads are unavailable until the node rejoins and reports Ready.
  • If gMSA is enabled on the source MKE 3 cluster, MKE 4 removes the legacy MKE 3 windows-gmsa components and deploys its own gMSA admission webhook; the spec.windows.gmsa.enabled parameter of the mke4.yaml configuration file is set automatically from the source configuration.
  • Limitations

To upgrade an MKE 3 cluster, use the mkectl upgrade command:

mkectl upgrade --hosts-path <path-to-hosts-yaml> \
  --mke3-admin-username <admin-username> \
  --mke3-admin-password <admin-password> \
  --external-address <external-address>\
  --config-out <path-to-desired-file-location>
Flag Detail
--dry-run Enables you to perform tests of your upgrade without having to perform an actual upgrade, with a resulting output of the the various checks and the upgrade configurations. By default, the source MKE 3 and target MKE 4 configurations are printed to stdio.
--output Enables upgrade results to be saved to a folder, created in the mke-upgrade format.
  • If the folder already exists, you will be prompted to overwrite it.
  • If the flag is used without a path, the action will default to the XDG directory for your OS. Otherwise, you can indicate a specific folder by entering --output=<path>. Use the help command to learn the exact output path.
  • The directory created for the output files will change depending on the command you ran. Upgrades will use mke-upgrade/.
  • Depending on the construction of the command you run and the context, the output folder will contain a combination of the following:
    • An mke3.toml configuration file
    • An mke4.yaml configuration file
    • Pre-upgrade checks, such as node health, cluster health, and version checks
    • Feature compatibility checks
    • Run summary
--force Enables you to continue to update without prompting for confirmation.

Info

You can pair the --dry-run flag with the --output flag to save the output results for an upgrade dry run to a specified folder.

Offline upgrade#

To perform an offline upgrade from MKE 3 to MKE 4, prepare your environment as described in Offline installation, and add the following flags to the mkectl upgrade command:

  • --image-registry=<registry_full_path>

    Sets your registry address with a project path that contains your MKE 4 images. For example, private-registry.example.com:8080/mke.

    The setting must not end with a slash /. The port is optional.

  • --chart-registry=oci://<registry_full_path>

    Sets your registry address with a project path that contains your MKE 4 helm charts in OCI format. For example, oci://private-registry.example.com:8080/mke.

    The setting must always start with oci://, and it must not end with a slash / .

    If you uploaded the bundle as previously described, the registry address and path will be the same for chart and image registry, with the only difference being the oci:// prefix in the chart registry URL.

  • --chart-registry-ca-file=<path/to/ca-cert.pem>

    Sets the path to the PEM encoded certificate of the Certificate Authority that issued chart registry TLS certificates. Optional. Must be provided if registry is using TLS certs issued by a non-publicly trusted CA.

  • --image-registry-ca-file=<path/to/ca-cert.pem>

    Sets the path to the PEM encoded certificate of the Certificate Authority that issued image registry TLS certificates. Optional. Must be provided if registry is using TLS certs issued by a non-publicly trusted CA.

  • --mke3-airgapped=true

    Indicates that your environment is air-gapped.

Warning

You cannot change the registries once the upgrade has completed.

Upgrade failure#

Note

In the event of upgrade failure, a support bundle automatically generates in the manager node. You can use this support bundle to help troubleshoot the upgrade procedure.

Example support bundle:

ERR MKE3 upgrade failed due to error: failed to run step [Migrate MKE 3 Nodes]: unable to add the first group of managers to k0s cluster: unable to build and apply after adding nodes: k0sctl apply failed while adding nodes: controller did not reach ready state: context deadline exceeded
readiness endpoint reports "": command failed: client exec: ssh session wait: Process exited with status 1, support dump available in /home/<user-name>/.local/state/mke4-dsinfo/mke4-dsinfo-mke3-upgrade-2025-09-17T20-52-22Z.zip
Error: failed to upgrade MKE to version v4.1.2: failed to run step [Migrate MKE 3 Nodes]: unable to add the first group of managers to k0s cluster: unable to build and apply after adding nodes: k0sctl apply failed while adding nodes: controller did not reach ready state: context deadline exceeded
readiness endpoint reports "": command failed: client exec: ssh session wait: Process exited with status 1

In the event of an upgrade failure, the upgrade process rolls back, restoring the MKE 3 cluster to its original state.

WARN[0096] Initiating rollback because of upgrade failure. upgradeErr = aborting upgrade due to signal interrupt
INFO[0096] Initiating rollback of MKE to version: 3.9.3
INFO[0096] Step 1 of 2: [Rollback Upgrade Tasks]
INFO[0096] Resetting k0s using k0sctl ...
INFO[0106] ==> Running phase: Connect to hosts
INFO[0106] [ssh] 54.151.30.20:22: connected
INFO[0106] [ssh] 54.215.145.126:22: connected
INFO[0106] ==> Running phase: Detect host operating systems
INFO[0106] [ssh] 54.151.30.20:22: is running Ubuntu 22.04.5 LTS
INFO[0106] [ssh] 54.215.145.126:22: is running Ubuntu 22.04.5 LTS
INFO[0106] ==> Running phase: Acquire exclusive host lock
INFO[0107] ==> Running phase: Prepare hosts
INFO[0107] ==> Running phase: Gather host facts
INFO[0107] [ssh] 54.151.30.20:22: using ip-172-31-8-69.us-west-1.compute.internal as hostname
INFO[0107] [ssh] 54.215.145.126:22: using ip-172-31-48-46.us-west-1.compute.internal as hostname
INFO[0107] [ssh] 54.151.30.20:22: discovered ens5 as private interface
INFO[0107] [ssh] 54.215.145.126:22: discovered ens5 as private interface
INFO[0107] [ssh] 54.151.30.20:22: discovered 172.31.8.69 as private address
INFO[0107] [ssh] 54.215.145.126:22: discovered 172.31.48.46 as private address
INFO[0107] ==> Running phase: Gather k0s facts
INFO[0108] [ssh] 54.215.145.126:22: found existing configuration
INFO[0108] [ssh] 54.215.145.126:22: is running k0s controller+worker version v1.31.1+k0s.1
WARN[0108] [ssh] 54.215.145.126:22: the controller+worker node will not schedule regular workloads without toleration for node-role.kubernetes.io/master:NoSchedule unless 'noTaints: true' is set
INFO[0108] [ssh] 54.215.145.126:22: listing etcd members
INFO[0110] [ssh] 54.151.30.20:22: is running k0s worker version v1.31.1+k0s.1
INFO[0110] [ssh] 54.215.145.126:22: checking if worker ip-172-31-8-69.us-west-1.compute.internal has joined
INFO[0110] ==> Running phase: Reset workers
INFO[0111] [ssh] 54.151.30.20:22: reset
INFO[0111] ==> Running phase: Reset controllers
INFO[0114] [ssh] 54.215.145.126:22: reset
INFO[0114] ==> Running phase: Reset leader
INFO[0114] [ssh] 54.215.145.126:22: reset
INFO[0114] ==> Running phase: Reload service manager
INFO[0114] [ssh] 54.151.30.20:22: reloading service manager
INFO[0114] [ssh] 54.215.145.126:22: reloading service manager
INFO[0115] ==> Running phase: Release exclusive host lock
INFO[0115] ==> Running phase: Disconnect from hosts
INFO[0115] ==> Finished in 8s
INFO[0125] Running etcd cleanup service ...
INFO[0128] MKE3 etcd directories successfully cleaned up ...
INFO[0128] Restoring etcd from the previously taken system backup ...
INFO[0128] Successfully restored etcd with output:
INFO[0128] Deploying etcd server ...
INFO[0129] Successfully deployed the etcd server with output: fb09c3e5e514d9ffe03a3df4bc461c29a695cf73d703ace5294702b7023021af
INFO[0129] Waiting for etcd to be healthy ...
INFO[0131] etcd health check succeeded!
INFO[0178] [Rollback Upgrade Tasks] Completed
INFO[0178] Step 2 of 2: [Rollback Pre Upgrade Tasks]
INFO[0178] [Rollback Pre Upgrade Tasks] Completed
INFO[0178] Rollback to MKE version 3.9.3 completed successfully ...
FATA[0178] Upgrade failed due to error: aborting upgrade due to signal interrupt

Info

A failed upgrade can leave behind an empty or corrupted ~/.mke/mke.kubeconf file, which will block any subsequent upgrade attempts. To resolve this issue, manually delete the file:

rm -f ~/.mke/mke.kubeconf

Following this action, during the next upgrade, the system will automatically generate a valid configuration.

Recover from an interrupted upgrade without rollback#

The automatic rollback runs only while the mkectl upgrade process is running. If the process is killed, or if the machine that runs mkectl loses power or network connectivity, the upgrade stops and does not roll back. The cluster is then left in a partial state: the MKE 3 components are stopped, and k0s is running on some or all of the nodes.

Warning

Do not run mkectl apply or mkectl upgrade against a cluster in this state. mkectl apply replaces the encryption key that the upgrade migrated from MKE 3, which makes all existing Secrets unreadable.

Note

If the cluster has Windows worker nodes, contact Mirantis Support before you perform this procedure. The automatic rollback restores the Windows container runtime, and this procedure does not.

To recover, uninstall MKE 4, restore MKE 3 from the backup that you created before the upgrade, and then upgrade again:

  1. On the machine where you run mkectl, create an mke4.yaml configuration file. An interrupted upgrade does not write this file, and thus if you need to create it:

    mkectl init > mke4.yaml

    In the mke4.yaml configuration file, make the following edits:

    • Under spec.hosts, list every cluster node with its role, SSH user, and SSH key path.
    • Set spec.apiServer.externalAddress to the external address that you passed to mkectl upgrade.

  2. Uninstall MKE 4 from all nodes:

    mkectl reset --force -f mke4.yaml

  3. Delete the MKE 4 kubeconfig file that the interrupted upgrade wrote. If this file exists, mkectl upgrade treats the cluster as an MKE 4 cluster and does not perform the upgrade.

    rm -f ~/.mke/mke.kubeconf

  4. On an MKE 3 manager node, remove the com.docker.ucp.agent-pause label from every Swarm node. The upgrade sets this label to stop the MKE 3 node agents, and neither the MKE 4 uninstall nor the MKE 3 restore removes it. If any node retains the label, the restore stalls at Waiting for <n> nodes to be ready.

    for node in $(docker node ls --format '{{.Hostname}}'); do
  docker node update --label-rm com.docker.ucp.agent-pause "$node"
done

  5. On the same manager node, uninstall the remaining MKE 3 components. The MKE 3 controller is no longer running, so pass --id force in place of the instance ID:

    docker container run --rm -i --log-driver none \
  -v /var/run/docker.sock:/var/run/docker.sock \
  mirantis/ucp:<mke3-version> uninstall-ucp --id force

  6. On the same manager node, restore MKE 3 from the backup. Use the MKE 3 version that created the backup, and pass a --san option for every external DNS name and IP address that clients use to reach the cluster, such as the load balancer address. The restore regenerates the API server certificate, and --san ensures that the new certificate is valid for these addresses.

    docker container run --rm -i --log-driver none --name ucp \
  -v /var/run/docker.sock:/var/run/docker.sock \
  mirantis/ucp:<mke3-version> restore \
  --san <external-address> < <mke3-backup>.tar

    For more information, refer to the MKE 3 Restore MKE documentation.

  7. Run mkectl upgrade again with the same options that you used for the original upgrade.