Restore#

To restore an MKE 4 cluster, you bootstrap a new cluster fromm a backup and then bring the rest of the cluster back online.

Restore scenario	Description
Restore to the same configuration	Recreate the cluster with the same nodes, roles, and configuration as the backup. Commonly used to rollback after an unwanted change.
Restore to a different node configuration	Restore onto nodes that differ from the original, such as different names, IP addresses, labels, or annotations.
Disaster recovery	Recover onto new infrastructure, as your original infrastructure is no longer available and the original cluster is irretrievable.

Warning

You must flatten the nodes that you use for disaster recovery, as they may have previously been a part of any MKE 4 or k0s cluster.
You must configure the bootstrap node appropriately before you attempt disaster recovery, meeting the same fs.inotify limits required of every cluster node. Refer to System requirements for the required values, and apply them immediately with sudo sysctl <parameter>=<value> for each one, in addition to persisting them as described.
Verify that the bootstrap node has enough free space in the directory where k0s restore unpacks the archive. The default directory is /tmp. To use a different directory, pass --temp-dir, or set spec.backup.tempDir in the k0s configuration before you create the backup. The required space equals the uncompressed size of the k0s archive in the backup. This archive includes the OCI image bundle directory and can reach several GB on offline clusters.

Important

You cannot use mkectl restore with the backup that mkectl upgrade creates automatically, mke4preupgradesnapshot<timestamp>.tar. If the automatic rollback of a failed upgrade does not complete, restore the backup that you created before the upgrade. Use the same mkectl version that created that backup.
Do not change the DNS name for the load balancer while performing disaster recovery. If you opt to deploy a new load balancer, ensure that it uses the same DNS name as the one in use by the original cluster.

Info

During disaster recovery, the original nodes are not accessible in the new environment. As such, because the entire cluster will initially be brought up on a single node, some of the containers and processes may encounter resource issues. You can resolve these issues either by using a highly resourced node or through operating system configuration.

Bootstrap a new cluster#

The first step in the restore process is to bootstrap a new cluster on a single node from the backup tar file:

Prepare a hosts.yaml file that contains the information needed to SSH into the bootstrap node:

  hosts:
  - ssh:
      address: <bootstrap-node-ip-address>
      user: <SSH-user-name>
      port: <ssh port>
      keyPath: <full-path-to-SSH-private-key>
    role: controller+worker

Restore the cluster:

mkectl restore -l debug --hosts-path <full-path-to-hosts.yaml-file-on-node-from-which-you-run-mkectl> --name <backup-tar-file>

If the archive is encrypted, add --passphrase. To verify the archive's integrity, add --md5 with the checksum you recorded when you created the backup:

mkectl restore --hosts-path <hosts-file> --name <backup-tar-file> \
  --passphrase - --md5 <checksum> --force \
  < /path/to/passphrase-file

If the archive contains a non-current configuration, also pass --accept-noncurrent-config.

mkectl runs the passphrase and checksum checks before it makes any change to the cluster:

Condition	Result
Archive not encrypted, `--passphrase` omitted	Restore proceeds.
Archive not encrypted, `--passphrase` specified	Restore fails. Re-run the command without `--passphrase`.
Archive encrypted, `--passphrase` omitted	Restore fails. For a scheduled backup, the error names the Secret and key identifier of the retained passphrase to use. For a manual backup, it asks for the passphrase supplied when the archive was created.
Archive encrypted, passphrase incorrect	Restore fails to decrypt the archive.

Backups require the matching mkectl build

Restore requires the matching mkectl build that created the backup. Store each mkectl binary with the backups it creates, and retain it for as long as you retain those backups.

The mkectl restore command prompts you to confirm before it bootstraps the cluster. If the archive contains a configuration that was not current at backup time, the command also displays the configuration provenance and prompts you to accept that configuration.

Each prompt has its own flag:

--force skips the bootstrap confirmation.
--accept-noncurrent-config accepts the non-current configuration.

For unattended restore, or when you read the passphrase from standard input with --passphrase -, pass both flags:

mkectl restore --hosts-path <hosts-file> --name <backup-tar-file> \
  --passphrase - --force --accept-noncurrent-config \
  < /path/to/passphrase-file

For details, refer to Configuration snapshot fallback.

Restore an archive from an earlier MKE 4 release with the mkectl binary that created it.

Authenticated registries

If the cluster pulls from a registry that requires authentication, pass the image and chart registry credentials to the mkectl restore command as well. Because the cluster does not exist during the restore, mkectl cannot read the credentials from it. For more information, refer to Supply the credentials during restore.

Add additional cluster nodes#

After you restore the bootstrap node, join the remaining nodes:

Prepare a hosts file that lists the remaining control plane nodes and the Linux worker nodes. Do not include the bootstrap node or the Windows worker nodes. The --scope flag of the mkectl node add command selects which entries in the file each run acts on, so you can use the same file for steps 2 and 3.

hosts:
  - role: controller+worker
    ssh:
      address: <controller-2-ip-address>
      user: <SSH-user-name>
      port: <ssh port>
      keyPath: <full-path-to-SSH-private-key>
  - role: controller+worker
    ssh:
      address: <controller-3-ip-address>
      user: <SSH-user-name>
      port: <ssh port>
      keyPath: <full-path-to-SSH-private-key>
  - role: worker
    ssh:
      address: <worker-1-ip-address>
      user: <SSH-user-name>
      port: <ssh port>
      keyPath: <full-path-to-SSH-private-key>

Join the control plane nodes:
```
mkectl node add --hosts-path <hosts-file> --scope controller+worker
```
The control plane nodes join one at a time. Because mkectl takes a cluster backup before and after adding control planes, which cause changes to etcd membership, you must pass either --passphrase or --no-passphrase. To skip the backups, pass --skip-snapshot instead. For more information, refer to Backups.

Join the Linux worker nodes:

mkectl node add --hosts-path <hosts-file> --scope worker

If the cluster has Windows worker nodes, join them last with the mkectl apply command, because mkectl node add does not support Windows nodes.

Warning

Do not apply the mke4.yaml file that mkectl restore writes. That file lists only the hosts from the restore hosts file, and mkectl apply fails when the control plane nodes in spec.hosts do not match those of the cluster, as mkectl apply does not add or remove control plane nodes. Use the output of mkectl config get instead.
1. Save the current cluster configuration, which lists every node that has joined the cluster:
```
mkectl config get > mke4.yaml
```
2. Add the Windows worker nodes to the end of the spec.hosts list in mke4.yaml.
3. Run the mkectl apply command:
```
mkectl apply -f mke4.yaml
```

Note

The mkectl config get command returns the configuration from the backup, as restored, with the joined nodes added. Configuration changes made after the backup was taken are not included and must be re-applied.

Rebalance the nodes#

Info

Rebalancing is not a factor for clusters that have a single controller node.

After you restore an MKE 4 cluster, join all of the remaining control plane nodes with a single mkectl node add command. The command automatically redistributes the MKE system components across the cluster, so no manual rebalancing is required. Joining the nodes in one invocation ensures that the redistribution occurs only once. For more information, refer to Workload redistribution.