Deploy a MSR cache with Swarm¶

This example guides you in deploying a MSR cache, assuming that you’ve got a MSR deployment up and running. It also assumes that you’ve provisioned multiple nodes and joined them into a swarm.

The MSR cache is going to be deployed as a Docker service, so that Docker automatically takes care of scheduling and restarting the service if something goes wrong.

We’ll manage the cache configuration using a Docker configuration, and the TLS certificates using Docker secrets. This allows you to manage the configurations securely and independently of the node where the cache is actually running.

Dedicate a node for the cache¶

To make sure the MSR cache is performant, it should be deployed on a node dedicated just for it. Start by labelling the node where you want to deploy the cache, so that you target the deployment to that node.

Use SSH to log in to a manager node of the swarm where you want to deploy the MSR cache. If you’re using MKE to manage that swarm, use a client bundle to configure your Docker CLI client to connect to the swarm.

docker node update --label-add dtr.cache=true <node-hostname>

Prepare the cache deployment¶

Create a file structure, as illustrated below:

├── docker-stack.yml    # Stack file to deploy cache with a single command
├── config.yml          # The cache configuration file
└── certs
    ├── cache.cert.pem  # The cache public key certificate
    ├── cache.key.pem   # The cache private key
    └── dtr.cert.pem    # MSR CA certificate

Add the following content to each of the two YAML files, docker-stack.yml and config.yml:

docker-stack.yml

version: "3.3"
services:
cache:
   image: mirantis/dtr-content-cache:2.8.2
   entrypoint:
      - /start.sh
      - "/config.yml"
   ports:
      - 443:443
   deploy:
      replicas: 1
      placement:
      constraints: [node.labels.dtr.cache == true]
      restart_policy:
      condition: on-failure
   configs:
      - config.yml
   secrets:
      - dtr.cert.pem
      - cache.cert.pem
      - cache.key.pem
configs:
config.yml:
   file: ./config.yml
secrets:
dtr.cert.pem:
   file: ./certs/dtr.cert.pem
cache.cert.pem:
   file: ./certs/cache.cert.pem
cache.key.pem:
   file: ./certs/cache.key.pem

config.yml

version: 0.1
log:
level: info
storage:
delete:
   enabled: true
filesystem:
   rootdirectory: /var/lib/registry
http:
addr: 0.0.0.0:443
secret: generate-random-secret
host: https://<cache-url>
tls:
   certificate: /run/secrets/cache.cert.pem
   key: /run/secrets/cache.key.pem
middleware:
registry:
      - name: downstream
      options:
         blobttl: 24h
         upstreams:
            - https://<msr-url>:<msr-port>
         cas:
            - /run/secrets/dtr.cert.pem

cache.cert.pem

Add the public key certificate for the cache here. If the certificate has been signed by an intermediate certificate authority, append its public key certificate at the end of the file.

cache.key.pem

Add the unencrypted private key for the cache here.

dtr.cert.pem

The cache communicates with MSR using TLS. If you’ve customized MSR to use TLS certificates issued by a globally trusted certificate authority, the cache automatically trusts MSR.</p>

But if you’re using the default MSR configuration, or MSR is using TLS certificates signed by your own certificate authority, you need to configure the cache to trust MSR.</p>

Add the MSR CA certificate to the certs/dtr.cert.pem file. You can do this by running:

curl -sk https://<dtr-url>/ca > certs/dtr.cert.pem

Next, add content to the three cert pem files, as described.

pem file	Content to add
cache.cert.pem	Add the public key certificate for the cache. If the certificate has been signed by an intermediate certificate authority, append its public key certificate at the end of the file.
cache.key.pem	Add the unencrypted private key for the cache.
dtr.cert.pem	The cache communicates with MSR using TLS. If you’ve customized MSR to use TLS certificates issued by a globally trusted certificate authority, the cache automatically trusts MSR. If, though, you are using the default MSR configuration, or MSR is using TLS certificates signed by your own certificate authority, you need to configure the cache to trust MSR. To do this, add the MSR CA certificate to the certs/dtr.cert. pem file. curl -sk https://<msr-url>/ca > certs/dtr.cert.pem

With this configuration, the cache fetches image layers from MSR and keeps a local copy for 24 hours. After that, if a user requests that image layer, the cache fetches it again from MSR.

The cache is configured to persist data inside its container. If something goes wrong with the cache service, Docker automatically redeploys a new container, but previously cached data is not persisted. You can customize the storage parameters, if you want to store the image layers using a persistent storage backend.

Also, the cache is configured to use port 443. If you’re already using that port in the swarm, update the deployment and configuration files to use another port. Don’t forget to create firewall rules for the port you choose.

Deploy the cache¶

Now that everything is set up, you can deploy the cache by running:

docker stack deploy --compose-file docker-stack.yml dtr-cache

You can check if the cache has been successfully deployed by running:

docker stack ps dtr-cache

Docker should show the dtr-cache stack is running.

Register the cache with MSR¶

Now that you’ve deployed a cache, you need to configure MSR to know about it. This is done using the POST /api/v0/content_caches API. You can use the MSR interactive API documentation to use this API.

In the MSR web UI, click the top-right menu, and choose API docs.

Navigate to the POST /api/v0/content_caches line and click it to expand. In the body field include:

{
  "name": "region-asia",
  "host": "https://<cache-url>:<cache-port>"
}

Click the Try it out! button to make the API call.

Configure your user account¶

Now that you’ve registered the cache with MSR, users can configure their user profile to pull images from MSR or the cache.

In the MSR web UI, navigate to your Account, click the Settings tab, and change the Content Cache settings to use the cache you deployed.

If you need to set this for multiple users at the same time, use the /api/v0/accounts/{username}/settings API endpoint.

Now when you pull images, you’ll be using the cache.

Test that the cache is working¶

To validate that the cache is working as expected:

Push an image to MSR.
Make sure your user account is configured to use the cache.
Delete the image from your local system.
Pull the image from MSR.

To validate that the cache is actually serving your request, and to troubleshoot misconfigurations, check the logs for the cache service by running:

docker service logs --follow dtr-cache_cache

The most common causes of configuration are due to TLS authentication:

MSR not trusting the cache TLS certificates.
The cache not trusting MSR TLS certificates.
Your machine not trusting MSR or the cache.

When this happens, check the cache logs to troubleshoot the misconfiguration.

Clean up sensitive files¶

The certificates and private keys are now managed by Docker in a secure way. Don’t forget to delete sensitive files you’ve created on disk, like the private keys for the cache:

rm -rf certs