Using Docker Content Trust with a Remote MKE Cluster

For more advanced deployments, you may want to share one Mirantis Secure Registry across multiple Mirantis Kubernetes Engines. However, customers wanting to adopt this model alongside the Only Run Signed Images MKE feature, run into problems as each MKE operates an independent set of users.

Docker Content Trust (DCT) gets around this problem, since users from a remote MKE are able to sign images in the central MSR and still apply runtime enforcement.

In the following example, we will connect MSR managed by MKE cluster 1 with a remote MKE cluster which we are calling MKE cluster 2, sign the image with a user from MKE cluster 2, and provide runtime enforcement within MKE cluster 2. This process could be repeated over and over, integrating MSR with multiple remote MKE clusters, signing the image with users from each environment, and then providing runtime enforcement in each remote MKE cluster separately.

Note

Before attempting this guide, familiarize yourself with Docker Content Trust and Only Run Signed Images on a single MKE. Many of the concepts within this guide may be new without that background.

Prerequisites

  • Cluster 1, running MKE 3.5.x or later, with an MSR 2.9.x or later deployed within the cluster.

  • Cluster 2, running MKE 3.5.x or later, with no MSR node.

  • Nodes on Cluster 2 need to trust the Certificate Authority which signed MSR’s TLS Certificate. This can be tested by logging on to a cluster 2 virtual machine and running curl https://msr.example.com.

  • The MSR TLS Certificate needs to be properly configured, ensuring that the Loadbalancer/Public Address field has been configured, with this address included within the certificate.

  • A machine with MCR 20.10.x or later installed, as this contains the relevant docker trust commands.

Registering MSR with a remote Mirantis Kubernetes Engine

As there is no registry running within cluster 2, by default MKE will not know where to check for trust data. Therefore, the first thing we need to do is register MSR within the remote MKE in cluster 2. When you normally install MSR, this registration process happens by default to a local MKE, or cluster 1.

Note

The registration process allows the remote MKE to get signature data from MSR, however this will not provide Single Sign On (SSO). Users on cluster 2 will not be synced with cluster 1’s MKE or MSR. Therefore when pulling images, registry authentication will still need to be passed as part of the service definition if the repository is private. See the Kubernetes example.

To add a new registry, retrieve the Certificate Authority (CA) used to sign the MSR TLS Certificate through the MSR URL’s /ca endpoint.

$ curl -ks https://msr.example.com/ca > dtr.crt

Next, convert the MSR certificate into a JSON configuration file for registration within the MKE for cluster 2.

You can find a template of the dtr-bundle.json below. Replace the host address with your MSR URL, and enter the contents of the MSR CA certificate between the new line commands \n and \n.

Note

JSON Formatting

Ensure there are no line breaks between each line of the MSR CA certificate within the JSON file. Use your favorite JSON formatter for validation.

$ cat dtr-bundle.json
{
  "hostAddress": "msr.example.com",
  "caBundle": "-----BEGIN CERTIFICATE-----\n<contents of cert>\n-----END CERTIFICATE-----"
}

Now upload the configuration file to cluster 2’s MKE through the MKE API endpoint, /api/config/trustedregistry_. To authenticate against the API of cluster 2’s MKE, we have downloaded an MKE client bundle, extracted it in the current directory, and will reference the keys for authentication.

$ curl --cacert ca.pem --cert cert.pem --key key.pem \
    -X POST \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -d @dtr-bundle.json \
    https://cluster2.example.com/api/config/trustedregistry_

Navigate to the MKE web interface to verify that the JSON file was imported successfully, as the MKE endpoint will not output anything. Select Admin > Admin Settings > Mirantis Secure Registry. If the registry has been added successfully, you should see the MSR listed.

Additionally, you can check the full MKE configuration file within cluster 2’s MKE. Once downloaded, the ucp-config.toml file should now contain a section called [registries]

$ curl --cacert ca.pem --cert cert.pem --key key.pem https://cluster2.example.com/api/ucp/config-toml > ucp-config.toml

If the new registry isn’t shown in the list, check the ucp-controller container logs on cluster 2.

Signing an image in MSR

We will now sign an image and push this to MSR. To sign images we need a user’s public private key pair from cluster 2. It can be found in a client bundle, with key.pem being a private key and cert.pem being the public key on an X.509 certificate.

First, load the private key into the local Docker trust store (~/.docker/trust). The name used here is purely metadata to help keep track of which keys you have imported.

docker trust key load --name cluster2admin key.pem
Loading key from "key.pem"...
Enter passphrase for new cluster2admin key with ID a453196:
Repeat passphrase for new cluster2admin key with ID a453196:
Successfully imported key from key.pem

Next initiate the repository, and add the public key of cluster 2’s user as a signer. You will be asked for a number of passphrases to protect the keys. Keep note of these passphrases, and see [Docker Content Trust documentation] (/engine/security/trust/trust_delegation/#managing-delegations-in-a-notary-server) to learn more about managing keys.

docker trust signer add --key cert.pem cluster2admin msr.example.com/admin/trustdemo
Adding signer "cluster2admin" to msr.example.com/admin/trustdemo...
Initializing signed repository for msr.example.com/admin/trustdemo...
Enter passphrase for root key with ID 4a72d81:
Enter passphrase for new repository key with ID dd4460f:
Repeat passphrase for new repository key with ID dd4460f:
Successfully initialized "msr.example.com/admin/trustdemo"
Successfully added signer: cluster2admin to msr.example.com/admin/trustdemo

Finally, sign the image tag. This pushes the image up to MSR, as well as signs the tag with the user from cluster 2’s keys.

docker trust sign msr.example.com/admin/trustdemo:1
Signing and pushing trust data for local image msr.example.com/admin/trustdemo:1, may overwrite remote trust data
The push refers to repository [dtr.olly.dtcntr.net/admin/trustdemo]
27c0b07c1b33: Layer already exists
aa84c03b5202: Layer already exists
5f6acae4a5eb: Layer already exists
df64d3292fd6: Layer already exists
1: digest: sha256:37062e8984d3b8fde253eba1832bfb4367c51d9f05da8e581bd1296fc3fbf65f size: 1153
Signing and pushing trust metadata
Enter passphrase for cluster2admin key with ID a453196:
Successfully signed msr.example.com/admin/trustdemo:1

Within the MSR web interface, you should now be able to see your newly pushed tag with the Signed text next to the size.

You could sign this image multiple times if required, whether it’s multiple teams from the same cluster wanting to sign the image, or you integrating MSR with more remote MKEs so users from clusters 1, 2, 3, or more can all sign the same image.

Enforce Signed Image Tags on the Remote MKE

We can now enable Only Run Signed Images on the remote MKE. To do this, login to cluster 2’s MKE web interface as an admin. Select Admin > Admin Settings > Docker Content Trust.

Finally we can now deploy a workload on cluster 2, using a signed image from a MSR running on cluster 1. This workload could be a simple $ docker run, a Swarm Service, or a Kubernetes workload. As a simple test, source a client bundle, and try running one of your signed images.

source env.sh

docker service create msr.example.com/admin/trustdemo:1
nqsph0n6lv9uzod4lapx0gwok
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

docker service ls
ID                  NAME                    MODE                REPLICAS            IMAGE                                   PORTS
nqsph0n6lv9u        laughing_lamarr         replicated          1/1                 msr.example.com/admin/trustdemo:1

Troubleshooting

If the image is stored in a private repository within MSR, you need to pass credentials to the Orchestrator as there is no SSO between cluster 2 and MSR. See the relevant Kubernetes documentation for more details.

Example Errors

Image or trust data does not exist

image or trust data does not exist for msr.example.com/admin/trustdemo:1

This means something went wrong when initiating the repository or signing the image, as the tag contains no signing data.

Image did not meet required signing policy

Error response from daemon: image did not meet required signing policy

msr.example.com/admin/trustdemo:1: image did not meet required signing policy

This means that the image was signed correctly, however the user who signed the image does not meet the signing policy in cluster 2. This could be because you signed the image with the wrong user keys.

MSR URL must be a registered trusted registry

Error response from daemon: msr.example.com must be a registered trusted registry. See 'docker run --help'.

This means you have not registered MSR to work with a remote MKE instance yet, as outlined in Registering MSR with a remote Mirantis Kubernetes Engine.