Upgrade MSR¶

MSR uses semantic versioning. While downgrades are not supported, Mirantis supports upgrades according to the following rules:

When upgrading from one patch version to another, you can skip patch versions because no data migration is performed for patch versions.
When upgrading between minor versions, you cannot skip versions, however you can upgrade from any patch version of the previous minor version to any patch version of the current minor version.
When upgrading between major versions, make sure to upgrade one major version at a time - and also to upgrade to the earliest available minor version. It is strongly recommended that you first upgrade to the latest minor/patch version for your major version.

Description	From	To	Supported
patch upgrade	x.y.0	x.y.1	yes
skip patch version	x.y.0	x.y.2	yes
patch downgrade	x.y.2	x.y.1	no
minor upgrade	x.y.*	x.y+1.*	yes
skip minor version	x.y.*	x.y+2.*	no
minor downgrade	x.y.*	x.y-1.*	no
skip major version	x..	x+2..	no
major downgrade	x..	x-1..	no
major upgrade	x.y.z	x+1.0.0	yes
major upgrade skipping minor version	x.y.z	x+1.y+1.z	no

A few seconds of interruption may occur during the upgrade of a MSR cluster, so schedule the upgrade to take place outside of peak hours to avoid any business impacts.

Minor upgrade¶

Important

Only perform the MSR upgrade once any correlating upgrades to Mirantis Kubernetes Engine (MKE) and/or Mirantis Container Runtime (MCR) have completed.

Mirantis recommends the following upgrade sequence:

Before starting the MSR upgrade, confirm that:

The version of MKE in use is supported by the upgrade version of MSR.
The MKE and MSR backups are both recent.

A backup of current swarm state has been created.

To create a swarm state backup, perform the following from a MKE manager node:

ENGINE=$(docker version -f '{{.Server.Version}}')
systemctl stop docker
sudo tar czvf "/tmp/swarm-${ENGINE}-$(hostname -s)-$(date +%s%z).tgz" /var/lib/docker/swarm/
systemctl start docker

(if possible) A backup exists of the images stored by MSR, if it is configured to store images on the local filesystem or within an NFS store.

BACKUP_LOCATION=/example_directory/filename
# If local filesystem
sudo tar -cf ${BACKUP_LOCATION} -C /var/lib/docker/volumes/dtr-registry-${REPLICA_ID}
# If NFS store
sudo tar -cf ${BACKUP_LOCATION} -C /var/lib/docker/volumes/dtr-registry-nfs-${REPLICA_ID}

None of the MSR replica nodes are exhibiting time drift. To make this determination, review the kernel log timestamps for each of the nodes. If time drift is occurring, use clock synchronization (e.g., NTP) to keep node clocks in sync.
Local filesystems across MSR nodes are not exhibiting any disk storage issues.
Docker Content Trust in MKE is disabled.
All system requirements are met.

Step 1. Upgrade MSR to 2.7 if necessary¶

Confirm that you are running MSR 2.7.x. If you are still using an earlier version of MSR, upgrade your installation to MSR 2.7.13.

Step 2. Upgrade MSR¶

Pull the latest version of MSR:

docker pull mirantis/dtr:2.8.13

Confirm that at least 16GB RAM is available on the node on which you are running the upgrade. If the MSR node does not have access to the internet, follow the offline installation documentation to get the images.

Once you have the latest image on your machine (and the images on the target nodes, if upgrading offline), run the upgrade command.

Note

The upgrade command can be run from any available node, as MKE is aware of which worker nodes have replicas.

docker run -it --rm \
  mirantis/dtr:2.8.13 upgrade

By default, the upgrade command runs in interactive mode and prompts for any necessary information. If you are performing the upgrade on an existing replica, pass the --existing-replica-id flag.

The upgrade command will start replacing every container in your MSR cluster, one replica at a time. It will also perform certain data migrations. If anything fails or the upgrade is interrupted for any reason, rerun the upgrade command (the upgrade will resume from the point of interruption).

Step 3. Verify Upgrade Success¶

To confirm that the newly upgraded MSR environment is ready:

Make sure that all running MSR containers reflect the newly upgraded MSR version:
```
docker ps --filter name=dtr
```
Verify that the MSR web UI is accessible and operational.
Confirm push and pull functionality of Docker images to and from the registry

Ensure that the MSR metadata store is in good standing:

REPLICA_ID=$(docker inspect -f '{{.Name}}' $(docker ps -q -f name=dtr-rethink) | cut -f 3 -d '-')
docker run -it --rm --net dtr-ol \
   -v dtr-ca-$REPLICA_ID:/ca \
   dockerhubenterprise/rethinkcli:v2.3.0 $REPLICA_ID
# List problems in the cluster detected by the current node.
> r.db("rethinkdb").table("current_issues")
[]

Metadata Store Migration¶

When upgrading from 2.5 to 2.6, the system will run a metadatastoremigration job following a successful upgrade. This involves migrating the blob links for your images, which is necessary for online garbage collection. With 2.6, you can log into the MSR web interface and navigate to System > Job Logs to check the status of the metadatastoremigration job.

Garbage collection is disabled while the migration is running. In the case of a failed metadatastoremigration, the system will retry twice.

If the three attempts fail, it will be necessary to manually retrigger the metadatastoremigration job. To do this, send a POST request to the /api/v0/jobs endpoint:

curl https://<msr-external-url>/api/v0/jobs -X POST \
-u username:accesstoken -H 'Content-Type':'application/json' -d \
'{"action": "metadatastoremigration"}'

Alternatively, select API from the bottom left-side navigation panel of the MSR web interface and use the Swagger UI to send your API request.

Patch upgrade¶

A patch upgrade changes only the MSR containers and is always safer than a minor version upgrade. The command is the same as for a minor upgrade.

MSR cache upgrade¶

If you have previously deployed a cache, be sure to upgrade the node dedicated for your cache to keep it in sync with your upstream MSR replicas. This prevents authentication errors and other strange behaviors.

Download the vulnerability database¶

After upgrading MSR, it is necessary to redownload the vulnerability database.