If your MSR has a majority of unhealthy replicas, the one way to restore it to a working state is by restoring from an existing backup.
To restore MSR, you need to:
You need to restore MSR on the same MKE cluster where you’ve created the backup. If you restore on a different MKE cluster, all MSR resources will be owned by users that don’t exist, so you’ll not be able to manage the resources, even though they’re stored in the MSR data store.
When restoring, you need to use the same version of the mirantis/dtr
image that you’ve used when creating the update. Other versions are not
guaranteed to work.
Start by removing any MSR container that is still running:
docker run -it --rm \
docker/dtr:2.7.5 destroy \
--ucp-insecure-tls
If you had MSR configured to store images on the local filesystem, you can extract your backup:
sudo tar -xf {{ image_backup_file }} -C /var/lib/docker/volumes
If you’re using a different storage backend, follow the best practices recommended for that system.
You can restore the MSR metadata with the mirantis/dtr restore
command. This performs a fresh installation of MSR, and reconfigures it
with the configuration created during a backup.
Load your MKE client bundle, and run the following command, replacing the placeholders for the real values:
read -sp 'ucp password: ' UCP_PASSWORD;
This prompts you for the MKE password. Next, run the following to restore MSR from your backup. You can learn more about the supported flags in mirantis/dtr restore.
docker run -i --rm \
--env UCP_PASSWORD=$UCP_PASSWORD \
docker/dtr:2.7.5 restore \
--ucp-url <mke-url> \
--ucp-insecure-tls \
--ucp-username <mke-username> \
--ucp-node <hostname> \
--replica-id <replica-id> \
--dtr-external-url <msr-external-url> < {{ metadata_backup_file }}
Where:
<mke-url>
is the url you use to access MKE<mke-username>
is the username of a MKE administrator<hostname>
is the hostname of the node where you’ve restored the
images<replica-id>
the id of the replica you backed up<msr-external-url>
the url that clients use to access MSRIf you’re using NFS as a storage backend, also include
--nfs-storage-url
as part of your restore command, otherwise DTR is
restored but starts using a local volume to persist your Docker images.
Warning
When running DTR 2.5 (with experimental online garbage collection) and MSR
2.6.0 to 2.6.3, there is an issue with reconfiguring and restoring MSR with
--nfs-storage-url
which leads to erased tags. Make sure to back up
your MSR metadata before you proceed. To work around
the --nfs-storage-url
flag issue, manually create a storage volume on
each DTR node. To restore DTR from an existing
backup, use mirantis/dtr restore
with --dtr-storage-volume
and the
new volume.
See Restore to a Local NFS Volume for Docker’s recommended recovery strategy.
If you’re scanning images, you now need to download the vulnerability database.
After you successfully restore MSR, you can join new replicas the same way you would after a fresh installation. Learn more.