External storage

External storage

Configure MSR image storage

Configure your storage backend

By default MSR uses the local filesystem of the node where it is running to store your Docker images. You can configure MSR to use an external storage backend, for improved performance or high availability.

If your MSR deployment has a single replica, you can continue using the local filesystem for storing your Docker images. If your MSR deployment has multiple replicas, make sure all replicas are using the same storage backend for high availability. Whenever a user pulls an image, the MSR node serving the request needs to have access to that image.

MSR supports the following storage systems:

  • Local filesystem

    • NFS

    • Bind Mount

    • Volume

  • Cloud Storage Providers

    • Amazon S3

    • Microsoft Azure

    • OpenStack Swift

    • Google Cloud Storage

Note

Some of the previous links are meant to be informative and are not representative of MSR’s implementation of these storage systems.

To configure the storage backend, log in to the MSR web interface as an admin, and navigate to System > Storage.

The storage configuration page gives you the most common configuration options, but you have the option to upload a configuration file in .yml, .yaml, or .txt format.

Local filesystem

By default, MSR creates a volume named dtr-registry-<replica-id> to store your images using the local filesystem. You can customize the name and path of the volume by using mirantis/dtr install --dtr-storage-volume or mirantis/dtr reconfigure --dtr-storage-volume.

Important

When running 2.6.0 to 2.6.3 (with experimental online garbage collection), there is an issue with reconfiguring MSR with --nfs-storage-url which leads to erased tags. Make sure to back up your MSR metadata before you proceed. To work around the `–nfs-storage-url`` flag issue, manually create a storage volume on each MSR node. If MSR is already installed in your cluster, reconfigure MSR with the --dtr-storage-volume flag using your newly-created volume.

If you’re deploying MSR with high-availability, you need to use NFS or any other centralized storage backend so that all your MSR replicas have access to the same images.

To check how much space your images are utilizing in the local filesystem, SSH into the MSR node and run:

# Find the path to the volume
docker volume inspect dtr-registry-<replica-id>

# Check the disk usage
sudo du -hs \
$(dirname $(docker volume inspect --format '{{.Mountpoint}}' dtr-registry-<msr-replica>))

NFS

You can configure your MSR replicas to store images on an NFS partition, so that all replicas can share the same storage backend.

Cloud Storage

Amazon S3

MSR supports Amazon S3 or other storage systems that are S3-compatible like Minio.

Switching storage back ends

Switching storage back ends initializes a new metadata store and erases your existing tags. This helps facilitate online garbage collection. In earlier versions, MSR would subsequently start a tagmigration job to rebuild tag metadata from the file layout in the image layer store. This job has been discontinued for DTR 2.5.x (with garbage collection) and DTR 2.6, as your storage back end could get out of sync with your MSR metadata, like your manifests and existing repositories. As a best practice, MSR storage back ends and metadata should always be moved, backed up, and restored together.

The --storage-migrated flag in reconfigure lets you indicate the migration status of your storage data during a reconfigure. If you are not worried about losing your existing tags, you can skip the recommended steps below and perform a reconfigure.

Note

Starting with MSR 2.9.0, switching your storage back end does not initialize a new metadata store or erase your existing storage. MSR now requires the new storage back end to contain an exact copy of the prior configuration’s data. If this requirement is not met, the storage must be reinitialized using the --reinitialize-storage flag with the dtr reconfigure command, which reinitializes a new metadata store and erases your existing tags.

It is a best practice to always move, back up, and restore your storage back ends with your metadata.

Best practice for data migration

  1. Disable garbage collection by selecting “Never” under System > Garbage Collection, so blobs referenced in the backup that you create continue to exist. Make sure to keep it disabled while you’re performing the metadata backup and migrating your storage data.

  2. Back up your existing metadata.

  3. Migrate the contents of your current storage backend to the new one you are switching to. For example, upload your current storage data to your new NFS server.

  4. Restore MSR from your backup and specify your new storage backend.

  5. With MSR restored from your backup and your storage data migrated to your new backend, garbage collect any dangling blobs using the following API request:

    curl -u <username>:$TOKEN -X POST "https://<msr-url>/api/v0/jobs" -H "accept: application/json" -H "content-type: application/json" -d "{ \"action": \"onlinegc_blobs\" }"
    

    On success, you should get a 202 Accepted response with a job id and other related details. This ensures any blobs which are not referenced in your previously created backup get destroyed.

Alternative option for data migration

If you have a long maintenance window, you can skip some steps from above and do the following:

  1. Put MSR in “read-only” mode using the following API request:

    curl -u <username>:$TOKEN -X POST "https://<msr-url>/api/v0/meta/settings" -H "accept: application/json" -H "content-type: application/json" -d "{ \"readOnlyRegistry\": true }"
    

    On success, you should get a 202 Accepted response.

  2. Migrate the contents of your current storage backend to the new one you are switching to. For example, upload your current storage data to your new NFS server.

  3. Reconfigure MSR while specifying the --storage-migrated flag to preserve your existing tags.

Regarding previous versions…

  • Make sure to perform a backup before you change your storage backend when running DTR 2.5 (with online garbage collection) and 2.6.0-2.6.3.

  • Upgrade to DTR 2.6.4 and follow best practice for data migration to avoid the wiped tags issue when moving from one NFS server to another.

Configuring MSR for S3

You can configure MSR to store Docker images on Amazon S3, or other file servers with an S3-compatible API like Cleversafe or Minio.

Amazon S3 and compatible services store files in “buckets”, and users have permissions to read, write, and delete files from those buckets. When you integrate MSR with Amazon S3, MSR sends all read and write operations to the S3 bucket so that the images are persisted there.

Create a bucket on Amazon S3

Before configuring MSR you need to create a bucket on Amazon S3. To get faster pulls and pushes, you should create the S3 bucket on a region that’s physically close to the servers where MSR is running.

Start by creating a bucket. Then, as a best practice you should create a new IAM user just for the MSR integration and apply an IAM policy that ensures the user has limited permissions.

This user only needs permissions to access the bucket that you’ll use to store images, and be able to read, write, and delete files.

Here’s an example of a user policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::<bucket-name>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "arn:aws:s3:::<bucket-name>/*"
        }
    ]
}

Configure MSR

Once you’ve created a bucket and user, you can configure MSR to use it. In your browser, navigate to https://<msr-url. Select System > Storage.

Select the S3 option, and fill-in the information about the bucket and user.

Field

Description

Root directory

The path in the bucket where images are stored

AWS Region name

The region where the bucket is.

S3 bucket name

The name of the bucket to store the images.

AWS access key

The access key to use to access the S3 bucket. This can be left empty if you’re using an IAM policy.

AWS secret key

The secret key to use to access the S3 bucket. This can be left empty if you’re using an IAM policy.

Region endpoint

The endpoint name for the region you’re using.

There are also some advanced settings.

Field

Description

Signature version 4 auth

Authenticate the requests using AWS signature version 4.

Use HTTPS

Secure all requests with HTTPS, or make requests in an insecure way.

Skip TLS verification

Encrypt all traffic, but don’t verify the TLS certificate used by the storage backend.

Root CA certificate

The public key certificate of the root certificate authority that issued the storage backend certificate.

Once you click Save, MSR validates the configurations and saves the changes.

Configure your clients

If you’re using a TLS certificate in your storage backend that’s not globally trusted, you’ll have to configure all Mirantis Container Runtimes that push or pull from MSR to trust that certificate. When you push or pull an image MSR redirects the requests to the storage backend, so if clients don’t trust the TLS certificates of both MSR and the storage backend, they won’t be able to push or pull images.

And if you’ve configured MSR to skip TLS verification, you also need to configure all Mirantis Container Runtimes that push or pull from MSR to skip TLS verification. You do this by adding MSR to the list of insecure registries when starting Docker.

Supported regions

MSR supports the following S3 regions:

S3 Regions

us-east-1

us-east-1

us-east-2

us-west-1

us-west-2

eu-west-1

eu-west-2

eu-central-1

ap-south-1

ap-southeast-1

ap-southeast-2

ap-northeast-1

ap-northeast-2

sa-east-1

cn-north-1

us-gov-west-1

ca-central-1

Update your S3 settings on the web interface

When running 2.6.0 to 2.6.4 (with experimental online garbage collection), there is an issue with changing your S3 settings on the web interface which leads to erased metadata. Make sure to back up your MSR metadata before you proceed.

Restore MSR with S3

To restore MSR using your previously configured S3 settings, use restore with --dtr-use-default-storage to keep your metadata.

Configuring MSR for NFS

You can configure MSR to store Docker images in an NFS directory. Starting in DTR 2.6, changing storage backends involves initializing a new metadatastore instead of reusing an existing volume. This helps facilitate online garbage collection. See changes to NFS reconfiguration below if you have previously configured MSR to use NFS.

Before installing or configuring MSR to use an NFS directory, make sure that:

  • The NFS server has been correctly configured

  • The NFS server has a fixed IP address

  • All hosts running MSR have the correct NFS libraries installed

To confirm that the hosts can connect to the NFS server, try to list the directories exported by your NFS server:

showmount -e <nfsserver>

You should also try to mount one of the exported directories:

mkdir /tmp/mydir && sudo mount -t nfs <nfs server>:<directory> /tmp/mydir

Install MSR with NFS

One way to configure MSR to use an NFS directory is at install time:

docker run -it --rm mirantis/dtr:2.8.8 install \
  --nfs-storage-url <nfs-storage-url> \
  <other options>

Use the format nfs://<nfs server>/<directory> for the NFS storage URL. To support NFS v4, you can now specify additional options when running install with --nfs-storage-url.

When joining replicas to a MSR cluster, the replicas will pick up your storage configuration, so you will not need to specify it again.

Reconfigure MSR to use NFS

To support NFS v4, more NFS options have been added to the CLI.

Important

When running 2.6.0 to 2.6.3 (with experimental online garbage collection), there is an issue with reconfiguring and restoring MSR with --nfs-storage-url which leads to erased tags. Make sure to back up your MSR metadata before you proceed. To work around the --nfs-storage-url flag issue, manually create a storage volume. If MSR is already installed in your cluster, reconfigure MSR with the --dtr-storage-volume flag using your newly-created volume.

DTR 2.6.4

In DTR 2.6.4, a new flag, --storage-migrated, has been added to reconfigure which lets you indicate the migration status of your storage data during a reconfigure. Upgrade to 2.6.4 and follow the best practice for data migration in 2.6.4 when switching storage backends. The following shows you how to reconfigure MSR using an NFSv4 volume as a storage backend:

docker run --rm -it \
  mirantis/dtr:2.8.8 reconfigure \
  --ucp-url <mke_url> \
  --ucp-username <mke_username> \
  --nfs-storage-url <msr-registry-nf>
  --async-nfs
  --storage-migrated

To reconfigure MSR to stop using NFS storage, leave the --nfs-storage-url option blank:

docker run -it --rm mirantis/dtr:2.8.8 reconfigure \
  --nfs-storage-url ""