Docker Content Trust

Docker Content Trust

Docker Content Trust (DCT) allows you to securely push and pull images to a public or private registry. DCT provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.

Mirantis Kubernetes Engine (MKE) and Mirantis Secure Registry (MSR) use the same content trust model as Mirantis Container Runtime so you should understand how it works before reading this topic.

The following sections apply to how to use DCT specifically in Mirantis Container Runtime.


DCT uses delegations to allow you to control who can sign an image tag. A delegation holds a pair of private and public keys and could contain multiple pairs depending on your use case. To use delegations, it’s helpful to understand how users, teams, and delegations work together in MKE.

In MKE, all users are members of a team. MKE generates client bundles containing TLS certificates that are unique to a given user. A user can be a member of one or more MKE teams. MKE uses the public certificate from their client bundle to delegate the image signing process.

To enable image delegation, you must add the user’s public certificate to the targets/releases group. We suggest that you also add a role that identifies the user’s team, such as targets/developers.

Roles that are not part of the targets/releases group need to allow multiple signatures on a given image tag.

Signing images with multiple keys

Signing with multiple keys is only possible if the same image and tag are used for each key. You can create multiple delegations with different keys, allowing for multiple signatures on a tag but if you push the image to a different tag, repository, or registry, the signatures of the previous image aren’t available.

Additionally, if the digest of the tag changes, for example when you add layers or push a different image, the signatures will no longer be valid for the digest attached to the current tag. You can still pull the image by digest with those signatures but the digest would be associated with the new tag along with the signatures.

You can create delegations using notary or using docker trust, both of which are described in the following sections.

Adding delegations using notary

Notary is a tool for publishing and managing trusted collections of content. Publishers can digitally sign collections and consumers can verify the integrity and origin of content.

You can use notary to add delegations that enable image signing. Notary was designed to be a generic tool so when you use it for image signing, you can’t use the default values for the filepath or registry url parameters. If you do, some notary commands will fail, and others will succeed but will place files in the wrong directory.

The default notary command uses the following values.

notary -d ${HOME}/.notary -s https://notary-server:4443 ...

That’s the incorrect url, so a command like the one own in the following example will fail.

notary delegation add -p targets/releases --all-paths ~/developer.pem ~/qa.pem ~/devops.pem

You can prevent the error by either explicitly setting the values, or by using an environment variable.

# explicitly list the options for each command:

notary -d ${HOME}/.docker/trust -s delegation add -p targets/releases --all-paths ~/developer.pem ~/qa.pem ~/devops.pem

# use variables for NOTARY_OPTS, MSR_URL

notary ${NOTARY_OPTS} delegation add -p targets/releases --

In the following examples, we define the ${NOTARY_OPTS} and MSR_URL environment variables and then use them for every notary command.

# set environment variables to make the commands below portable
export MSR_URL=""
export NOTARY_OPTS="-d ${HOME}/.docker/trust -s https://${MSR_URL} --tlscacert ${HOME}/.docker/tls/${MSR_URL}/ca.crt"
export NAMESPACE="demo"
export REPO="dcttest"
export ROLE="demo"

# create directory and download CA from MSR for self-signed
mkdir -p ${HOME}/.docker/tls/${MSR_URL}
curl -sSLk https://${MSR_URL}/ca > ${HOME}/.docker/tls/${MSR_URL}/ca.crt

# remove data; local and remote (optional - WARNING! will remove all trust data remote and local)
notary ${NOTARY_OPTS} delete ${MSR_URL}/${NAMESPACE}/${REPO} --remote

# initialize repository in notary
notary ${NOTARY_OPTS} init ${MSR_URL}/${NAMESPACE}/${REPO}

# publish locally staged changes
notary ${NOTARY_OPTS} publish ${MSR_URL}/${NAMESPACE}/${REPO}

# rotate snapshot key and change it to server managed
notary ${NOTARY_OPTS} key rotate ${MSR_URL}/${NAMESPACE}/${REPO} snapshot --server-managed

# create delegation for 'targets/releases' role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/releases --all-paths cert.pem

# create delegation for 'targets/${ROLE}' role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/${ROLE} --all-paths cert.pem

# show delegations
notary ${NOTARY_OPTS} delegation list ${MSR_URL}/${NAMESPACE}/${REPO}

# load the key on the client
notary ${NOTARY_OPTS} key import key.pem

# enable DCT

# push image
docker push ${MSR_URL}/${NAMESPACE}/${REPO}:<tag>

When adding a delegation, you must also add the delegation for the targets/releases role and it’s recommended that you also add a role to indicate the user or team.

For example, if you add three teams – developer, qa, and devops – you must add them to the targets/releases role because Docker uses the targets/releases role to determine if there is a signature. You can also create a role for each key as shown in this example.

# create delegation for the targets/releases role for each of the keys

notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/releases --all-paths cert.pem

notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO}targets/releases --all-paths ~/developer.pem ~/qa.pem ~/devops.pem

# create delegation to the targets/developer role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/developer --all-paths ~/developer.pem

# create delegation to the targets/qa role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/qa --all-paths ~/qa.pem

# create delegation to the targets/devops role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/devops --all-paths ~/devops.pem

If everyone is pushing to the same tag, you end up with the same hash in targets/developer, targets/qa, and targets/devops, and then whomever signed last also signed the same hash into targets/releases.

If you don’t have a signature on targets/releases, you can’t pull the image. If you create individual roles for each, you get additional data about which signatures are actually in-place based off of the key that was used.

Adding delegations in PowerShell using notary

This example shows you how to do the same thing in PowerShell.

# set variables to make this copy and paste
$env:MSR_URL = ""
$env:NAMESPACE = "demo"
$env:REPO = "dcttest"
$env:ROLE = "demo"

notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delete ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} --remote

# initialize repository in notary
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} init ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO}

# publish locally staged changes
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} publish ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO}

# rotate snapshot key and change it to server managed
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} key rotate ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} snapshot --server-managed

# create delegation for 'targets/releases' role
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delegation add -p ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} targets/releases --all-paths cert.pem

# create delegation for 'targets/${ROLE}' role
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delegation add -p ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} targets/${env:ROLE} --all-paths cert.pem

# show delegations
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delegation list ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO}

# import key
notary -d ${HOME}\.docker\trust key import key.pem

# enabled DCT

Adding delegations using docker trust

You can also add delegations using docker trust.

The following commands assume that you already have a client bundle downloaded and extracted to the present working directory.

# login to MSR as the docker trust commands utilize your stored credentials for authentication
$ docker login
Username: admin

# set the proper environment variables for creating a delegation (only required if you want to not be prompted for passphrases for the DCT process)
$ export DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE="myrootpassphrase"
$ export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="myrepositorypassphrase"

# take your public key and add your user as a signer for the repository
$ docker trust signer add --key cert.pem admin

# verify that you see your user now listed as a signer
$ docker trust inspect --pretty

# set the proper environment variables to import the private key for
signing (only required if you want to not be prompted for passphrases for
the DCT process)
$ export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="myrepositorypassphrase"

# load your private key into your local storage so that it can
be used by Docker to sign
$ docker trust key load key.pem

# enable DCT on push

# retag an image to push to MSR
$ docker tag hello-world:latest

# push the image and sign
$ docker push

# verify that the image tag is now signed
$ docker trust inspect --pretty

Use Cases

Multiple key & signing with a multiple clients

The process of signing with multiple keys by multiple clients is possible by pulling and then pushing the image back to the registry under the same image tag. The user that is pulling and push the image back to the registry should have their key loaded locally in order to sign the image under their role.

Multiple key & signing with a single client

If you have loaded multiple keys to the trust store on a docker client, it will automatically sign with all keys that match a delegation. In the case where you need to only specify one key to sign an image, it is possible to specify the location of the Docker client’s config files (the default setting for ~/.docker). This allows you to isolate the configuration and imported keys to specifically sign with that key. So if you had three roles, developer, qa, and devops, you could create specific directories for each role. The follow example uses the developer role.

  1. Create a diretory for the developer role

    mkdir ~/.docker.developer
  2. Login, specifying the config path.

    docker --config ~/.docker.developer login
  3. Optional. Load the key into that specific directory:

    notary -d ~/.docker.developer/trust key import ~/developer.pem
  4. When you’re ready to push, sign with the key loaded into the trust directory for the developer role.

    DOCKER_CONTENT_TRUST=1 docker --config ~/.docker.developer push msr.fqdn/namespace/repo:tag
  5. Repeat for the other roles.

  6. You can verify the delegations using the following command.

    notary -s delegation list

Sign images after adding to a registry

You might want to sign an image that is already in a registry for a number of reasons – promotion policy, mirroring, manual release gates, multiple signatures, and so on. The following example shows how to sign an image that’s already in a registry. You can use $ docker push or $ docker trust sign in addition to the method outlined below that uses the notary cli.


This documentation assumes you have an image already pushed into MSR, that you have initialised and added a delegation, and that you have imported a local key.

  1. Get the digest ID and size using the manifest inspect command.

    -v is required because digest information isn’t available in a generic inspect command.

    $ docker manifest inspect -v
       "Ref": "",
       "Descriptor": {
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "digest": "sha256:fa037546bc7d548cf691131c07ff141a0ad7f2463336338eb82fe86822e437b2", # WE NEED THIS BIT
                "size": 1158, # AND WE NEED THIS BIT
                "platform": {
                         "architecture": "amd64",
                         "os": "linux"
       "SchemaV2Manifest": {
                "schemaVersion": 2,
                "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                "config": {
                         "mediaType": "application/vnd.docker.container.image.v1+json",
                         "size": 2545,
                         "digest": "sha256:1a1232d089075fd02d6c54c101e339009cc35f7c581572b506b8955e1bac19d1"
                "layers": [
                               "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                               "size": 2206542,
                               "digest": "sha256:8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6"
                               "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                               "size": 2735193,
                               "digest": "sha256:08578be31d4be43782cb20a6c6de9fe753f459485fd7ee0c8be9722ed67925fc"
                               "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                               "size": 648,
                               "digest": "sha256:a5d2afcf9fac34131270861ffc420b5e73b02ba7017b533dfd9390b1f2056b64"
                               "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                               "size": 15967740,
                               "digest": "sha256:b039e87a6cd1b0ebb4219c2ae5b4f6b966b24472ac8ff253d69f9ad7927316e4"
  2. Ensure your delegation is added to it’s own role in addition to the release role

    $ notary delegation list
  3. Sign the image for both roles.

    # The format of this command is:
    # $ notary addhash -p <GUN> <tag> <size of the manifest in bytes> --sha256 <sha256 hash of the manifest> -r targets/<role>
    GUN stands for Globally Unique Name
    # For example:
    $ notary addhash latest 1158 --sha256 fa037546bc7d548cf691131c07ff141a0ad7f2463336338eb82fe86822e437b2 -r targets/releases
    $ notary addhash latest 1158 --sha256 fa037546bc7d548cf691131c07ff141a0ad7f2463336338eb82fe86822e437b2 -r targets/admin
  4. Verify your setup using the following command.

    $ notary status
  5. Publish the changes.

    $ notary publish
  6. Check to make sure everything is signed correctly.

    $ docker trust inspect --pretty

Enabling DCT in Mirantis Container Runtime

DCT is controlled by MCR’s configuration file, daemon.json.

MCR verifies that images are signed before starting, which affects the docker run, docker pull, and docker build commmands, but does not affect Swarm services.


This configuration can be set on Linux machines only.

The content-trust flag is based around a mode variable instructing MCR whether to enforce signed images, and a trust-pinning variable instructing MCR which sources to trust.

Mode can take three variables:

  • Disabled - Verification is not active and the remainder of the content-trust related metadata will be ignored. This is the default value if mode is not specified.

  • Permissive - Verification will be performed, but only failures will be logged and remain unenforced. This configuration is intended for testing of changes related to content-trust. The results of the signature verification is displayed in the Mirantis Container Runtime’s daemon logs.

  • Enforced - DCT will be enforced and an image that cannot be verified successfully will not be pulled or run.

Here’s an example that sets the flag to enforced.

     "content-trust": {
         "mode": "enforced"

Every time the config is changed, the daemon needs to be reloaded.

Official Images

The root key for all* is pinned in MCR itself. Therefore you could limit MCR to only run official docker images.

"content-trust": {
   "trust-pinning": {
      "official-library-images": true
   "mode": "enforced"

User Signed Images

Mirantis Container Runtime provides two options for trust-pinning user-signed images.

  • Notary Canonical Root key ID (DCT root-key)

    An ID that describes just the root key used to sign a repository (or rather its respective keys).This is the root key on the host that originally signed the repository (for example a personal workstation).

    It can be retrieved from the workstation that signed the repository by running grep -r "root" ~/.docker/trust/private/, or by manually building notary: canonical-info and using the notary info <GUN> command (where GUN stands for Globally Unique Name). This canonical ID should have initiated multiple image repositories, for example mymsr/user1/image1 and mymsr/user1/image2.

  • Notary Root key ID (DCT Cert ID)

    An ID that describes the DCT root-key, but it is unique per repository. For example mymsr/user1/image1 and mymsr/usr1/image2 will have unique DCT Cert IDs. You can get the DCT CertID using a $ docker trust inspect command. You could use this method if every root repository was signed with a different host and therefore a different canonical root. Because DCT cert-id is more specific than the root ID, it takes priority or root ID.

Here’s an example daemon.json.

# Retrieving Cert ID
$ docker trust inspect | jq -r '.[].AdministrativeKeys[] | select(.Name=="Root") | .Keys[].ID'

# Retrieving Root ID
$ grep -r "root" ~/.docker/trust/private
/home/example/.docker/trust/private/0b6101527b2ac766702e4b40aa2391805b70e5031c04714c748f914e89014403.key:role: root

# Using a Canonical ID that has signed 2 repos (mymsr/user1/repo1 and mymsr/user1/repo2). Note you can use a Wildcard.

"content-trust": {
   "trust-pinning": {
      "root-keys": {
         "mymsr/user1/*": [
   "mode": "enforced"

# Using Cert Ids, by specifying 2 repositories by their DCT root ID. #Example for using this may be different MSRs or maybe because the #repository was initiated on different hosts, therefore having different #canonical IDs.

"content-trust": {
   "trust-pinning": {
      "cert-ids": {
         "mymsr/user1/repo1": [
         "mymsr/user2/repo1": [
   "mode": "enforced"

# Putting it all together

"content-trust": {
   "trust-pinning": {
      "official-library-images": true,
      "root-keys": {
         "mymsr/user1/*": [
      "cert-ids": {
         "mymsr/user2/repo1": [
   "mode": "enforced",
   # Enable offline environments if the engine can't communicate with the
   # registry
   "allow-expired-cached-trust-data": true



Here’s an example of how to automate delegation. It uses a new feature is available to be able to use NOTARY_AUTH as a base64 encoded username:password.

See Notary client configuration file for more details.

  1. Add MSR credentials

    export USERNAME="admin"
    export PASSWORD="docker123"
  2. Set a flag to create repos

    export CREATE_REPOS=false
  3. Set environment variables

    export MSR_URL=""
    export NAMESPACE="official"
    export REPO_LIST="alpine centos golang postgres nginx node redis ubuntu"
    export ROLE="demo"
    export USERNAME="username HERE"
    export PASSWORD="password HERE"
  4. Create notary automation variables

    export NOTARY_ROOT_PASSPHRASE="docker123"
    export NOTARY_TARGETS_PASSPHRASE="docker123"
    export NOTARY_SNAPSHOT_PASSPHRASE="docker123"
    export NOTARY_OPTS="-s https://${MSR_URL} -d ${HOME}/.docker/trust"
  5. Create repos (optional)

    if ${CREATE_REPOS}
      # create org
      curl -sk -X POST --header "Content-Type: application/json" --header "Accept: application/json" -u "${USERNAME}:${PASSWORD}" -d "{
          \"name\": \"official\",
          \"isOrg\": true,
          \"isAdmin\": true,
          \"isActive\": true
        }" "https://${MSR_URL}/enzi/v0/accounts"
      # create repos
      for i in ${REPO_LIST}
        curl -sk -X POST --header "Content-Type: application/json" --header "Accept: application/json" -u "${USERNAME}:${PASSWORD}" -d "{
          \"name\": \"${i}\",
          \"shortDescription\": \"Official image for ${i}\",
          \"longDescription\": \"Official image from ${i} from Docker Hub\",
          \"visibility\": \"public\"
        }" "https://${MSR_URL}/api/v0/repositories/official"
  6. Write script

    cat > /tmp/notary_expect.exp <<EOL
    #!/usr/bin/env expect -f
    eval spawn notary \$env(NOTARY_PARAMS)
    expect "Enter username: "
    send "\$env(USERNAME)\r"
    expect "Enter password: "
    send "\$env(PASSWORD)\r"
    expect eof
  7. Initialize the repos

    for i in ${REPO_LIST}
       echo -e "\ndelete local and remote data"
       NOTARY_PARAMS="${NOTARY_OPTS} delete ${MSR_URL}/${NAMESPACE}/${i} --remote" expect /tmp/notary_expect.exp
       echo -e "\ninitialize repo"
       NOTARY_PARAMS="${NOTARY_OPTS} init ${MSR_URL}/${NAMESPACE}/${i}" expect /tmp/notary_expect.exp
       echo -e "\npublish staged changes"
       NOTARY_PARAMS="${NOTARY_OPTS} publish ${MSR_URL}/${NAMESPACE}/${i}" expect /tmp/notary_expect.exp
       echo -e "\nrotate snapshot key"
       NOTARY_PARAMS="${NOTARY_OPTS} key rotate ${MSR_URL}/${NAMESPACE}/${i} snapshot --server-managed" expect /tmp/notary_expect.exp
       echo -e "\nadd cert to releases role"
       NOTARY_PARAMS="${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${i} targets/releases --all-paths cert.pem" expect /tmp/notary_expect.exp
       echo -e "\nadd cert to ${ROLE} role"
       NOTARY_PARAMS="${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${i} targets/${ROLE} --all-paths cert.pem" expect /tmp/notary_expect.exp
       echo -e "\nlisting delegations"
       notary ${NOTARY_OPTS} delegation list ${MSR_URL}/${NAMESPACE}/${i}
  8. Cleanup

    rm /tmp/notary_expect.exp


Here are common errors and how to work around them.

  1. Potential malicious behavior error

    Warning: potential malicious behavior - trust data has insufficient signatures for remote repository valid signatures did not meet threshold

    To fix this error, run notary delete on the client (without the remote flag). For example:

    notary -d ~/.docker/trust delete dtr.fqdn/namespace/repo
  2. Trusted root error

    Could not validate the path to a trusted root: failed to validate data with current trusted certificates

    You might see this error when attempting to push with a signature after deleting signing data from a repo. The error is raised because of a trusted_certificates directory under tuf that should not be there. 1.12 ignores it, but old versions use it for pinning.

    To fix this error, upgrade to 1.12 or remove the deprecated directory.

  3. Failed to validate data error

    {“level”:”debug”,”msg”:”notary.Middleware: Err getting targets: could not rotate trust to a new trusted root: failed to validate data with current trusted certificates”,”time”:”2016-11-09T22:25:18Z”}

    You will see this error when pulling and image after the DCT keys are reset and signed with a new key. MKE continues to store the old key, causing the error.

    To fix it, remove key data using the following example code.

    curl -v -X DELETE -H "Authorization: Bearer $(curl -sk -d '{"username":"username","password":"password"}' | jq -r .auth_token)"
  4. Missing valid signer error

    Error response from daemon: required at least one valid signer, none found

    This is the error that the user would actually see when trying to deploy a service where:

    1. There isn’t any trust data for the given tag

    2. The user’s delegation didn’t grant them access to the targets/releases role

    3. The trust data doesn’t match any users in MKE

    4. An image being pulled with DCT and if the keys are completely reset and signed with a different key, MKE continues to store that key in etcd. To fix it, remove the key data (Add endpoint to delete pinned notary trust data for a repoCLOSED ):

    To fix this ereror, run the following command, where is the Globally Unique Name (GUN) of your repo.

    docker exec -it -e ETCDCTL_API=3 ucp-kv etcdctl --endpoints --cert /etc/docker/ssl/cert.pem --key /etc/docker/ssl/key.pem --cacert /etc/docker/ssl/ca.pem del notary/v1/ --prefix