Restore#

To restore an MKE 4 cluster, you bootstrap a new cluster fromm a backup and then bring the rest of the cluster back online.

Restore scenario	Description
Restore to the same configuration	Recreate the cluster with the same nodes, roles, and configuration as the backup. Commonly used to rollback after an unwanted change.
Restore to a different node configuration	Restore onto nodes that differ from the original, such as different names, IP addresses, labels, or annotations.
Disaster recovery	Recover onto new infrastructure, as your original infrastructure is no longer available and the original cluster is irretrievable.

Warning

You must flatten the nodes that you use for disaster recovery, as they may have previously been a part of any MKE 4 or k0s cluster.
You must configure the bootstrap node appropriately before you attempt disaster recovery. For example, you may need to configuremax_user_instances and max_user_watches to a higher non-default value, as exemplified:
```
sudo sysctl fs.inotify.max_user_instances=1280
sudo sysctl fs.inotify.max_user_watches=655360
```

Important

Do not change the DNS name for the load balancer while performing disaster recovery. If you opt to deploy a new load balancer, ensure that it uses the same DNS name as the one in use by the original cluster.

Info

During disaster recovery, the original nodes are not accessible in the new environment. As such, because the entire cluster will initially be brought up on a single node, some of the containers and processes may encounter resource issues. You can resolve these issues either by using a highly resourced node or through operating system configuration.

Bootstrap a new cluster#

The first step in the restore process is to bootstrap a new cluster on a single node from the backup tar file:

Prepare a hosts.yaml file that contains the information needed to SSH into the bootstrap node:

  hosts:
  - ssh:
      address: <bootstrap-node-ip-address>
      user: <SSH-user-name>
      port: <ssh port>
      keyPath: <full-path-to-SSH-private-key>
    role: controller+worker

Restore the cluster:

mkectl restore -l debug --hosts-path <full-path-to-hosts.yaml-file-on-node-from-which-you-run-mkectl> --name <backup-tar-file>

Add additional cluster nodes#

Add the required additional nodes to the mke4.yaml configuration file that was generated as a result of bootstrapping a new cluster.

Tip

You can use the newly generated mke4.yaml configuration file to add the remaining nodes to the cluster, as well as to construct a one-to-one correspondence between nodes in the original cluster and nodes in the recreated cluster through the use of node labels.
Run the mkectl apply command.

Rebalance the nodes#

Info

Rebalancing is not a factor for clusters that have a single controller node.

After you restore a cluster, all MKE 4 system components run on the single bootstrap node and are not automatically rebalanced when the remaining control plane nodes rejoin as a result of the mkectl apply command. As such, although all Pods report Running, critical management components remain on the bootstrap node.

To restore High Availability and to enhance node use, once all of the control plane nodes have rejoined the cluster and reported as Ready, force a rollout to reschedule the components across the nodes:

kubectl rollout restart deployment -n mke
kubectl rollout restart deployment -n k0rdent