- Home
- Upgrades
- Upgrade an existing MKE 4 cluster
- Upgrade rollbacks
Upgrade rollbacks#
When an upgrade from a previous MKE 4 release fails, mkectl upgrade
automatically rolls the cluster back to the original version. You can disable
this behavior, and you can restore the cluster manually when the rollback
does not run to completion.
Automatic rollback#
The mkectl upgrade command runs a series of steps and reports each one in
its output. During the Pre Upgrade Backup step, the command creates a
pre-upgrade backup of the cluster. This backup exists only for the automatic
rollback, and you cannot restore it with mkectl restore.
If the upgrade fails, the cluster automatically rolls back to the original version. The rollback starts without prompting for confirmation, and rebuilds the cluster from the pre-upgrade backup.
-
Stops k0s on every node, and resets and reboots every controller node.
-
Restores etcd from the pre-upgrade backup on the first controller node, and then rejoins the remaining controller nodes to it.
-
Resets, reboots, and rejoins every worker node.
The mkectl upgrade command collects support bundles both before and after
the rollback. Use these support bundles to troubleshoot the failed upgrade.
Warning
The rollback interrupts all workloads on the cluster, including those that the failed step did not affect. To stop automatic rollback following a failed upgrade, disable the automatic rollback.
Disable the automatic rollback#
To leave the cluster as it is when an upgrade step fails, upgrade using the
--disable-rollback flag:
mkectl upgrade --disable-rollback
If a step fails, mkectl upgrade collects a support bundle, reports the
failed step, and exits without rolling back. The cluster may be left partially
upgraded, with some components on the new version and others on the original
version. The flag does not skip the pre-upgrade backup.
Use --disable-rollback when interrupting every workload costs more than a
partially upgraded cluster, for example on a production cluster where you
want to investigate a failure before anything else changes on the cluster.
After a failure, contact Mirantis Support and provide the support bundle. To return the cluster to the original version, restore it as described in Restore the cluster after a failed upgrade.
Restore the cluster after a failed upgrade#
Restore the cluster from the backup that you created before the upgrade, as described in Perform an MKE 4 to MKE 4+ Upgrade, when either of the following applies:
- You ran the upgrade with
--disable-rollbackand a step failed. - The upgrade stopped, and the automatic rollback did not start or did not
finish. A rollback can fail, for example, when the
mkectlprocess is terminated or the host that runs it loses power.
Use the mkectl binary that created the backup to restore it. For more information, refer to Backup.
Warning
Do not take either of these actions:
- Rerun the upgrade. A repeated upgrade does not reliably recover a partially upgraded cluster.
- Restore the pre-upgrade backup that
mkectl upgradecreated.mkectl restorecannot restore it.