Docker Content Trust¶

Delegations¶

DCT uses delegations to allow you to control who can sign an image tag. A delegation holds a pair of private and public keys and could contain multiple pairs depending on your use case. To use delegations, it’s helpful to understand how users, teams, and delegations work together in MKE.

In MKE, all users are members of a team. MKE generates client bundles containing TLS certificates that are unique to a given user. A user can be a member of one or more MKE teams. MKE uses the public certificate from their client bundle to delegate the image signing process.

To enable image delegation, you must add the user’s public certificate to the targets/releases group. We suggest that you also add a role that identifies the user’s team, such as targets/developers.

Roles that are not part of the targets/releases group need to allow multiple signatures on a given image tag.

Signing images with multiple keys¶

Signing with multiple keys is only possible if the same image and tag are used for each key. You can create multiple delegations with different keys, allowing for multiple signatures on a tag but if you push the image to a different tag, repository, or registry, the signatures of the previous image aren’t available.

Additionally, if the digest of the tag changes, for example when you add layers or push a different image, the signatures will no longer be valid for the digest attached to the current tag. You can still pull the image by digest with those signatures but the digest would be associated with the new tag along with the signatures.

You can create delegations using notary or using docker trust, both of which are described in the following sections.

Adding delegations using notary¶

Notary is a tool for publishing and managing trusted collections of content. Publishers can digitally sign collections and consumers can verify the integrity and origin of content.

You can use notary to add delegations that enable image signing. Notary was designed to be a generic tool so when you use it for image signing, you can’t use the default values for the filepath or registry url parameters. If you do, some notary commands will fail, and others will succeed but will place files in the wrong directory.

The default notary command uses the following values.

notary -d ${HOME}/.notary -s https://notary-server:4443 ...

That’s the incorrect url, so a command like the one own in the following example will fail.

notary delegation add -p https://msr.example.com/namespace/repo targets/releases --all-paths ~/developer.pem ~/qa.pem ~/devops.pem

You can prevent the error by either explicitly setting the values, or by using an environment variable.

# explicitly list the options for each command:

notary -d ${HOME}/.docker/trust -s https://msr.example.com delegation add -p https://msr.example.com/namespace/repo targets/releases --all-paths ~/developer.pem ~/qa.pem ~/devops.pem

# use variables for NOTARY_OPTS, MSR_URL

notary ${NOTARY_OPTS} delegation add -p https://msr.example.com/namespace/repo targets/releases --

In the following examples, we define the ${NOTARY_OPTS} and MSR_URL environment variables and then use them for every notary command.

# set environment variables to make the commands below portable
export MSR_URL="msr.example.com"
export NOTARY_OPTS="-d ${HOME}/.docker/trust -s https://${MSR_URL} --tlscacert ${HOME}/.docker/tls/${MSR_URL}/ca.crt"
export NAMESPACE="demo"
export REPO="dcttest"
export ROLE="demo"

# create directory and download CA from MSR for self-signed
mkdir -p ${HOME}/.docker/tls/${MSR_URL}
curl -sSLk https://${MSR_URL}/ca > ${HOME}/.docker/tls/${MSR_URL}/ca.crt

# remove data; local and remote (optional - WARNING! will remove all trust data remote and local)
notary ${NOTARY_OPTS} delete ${MSR_URL}/${NAMESPACE}/${REPO} --remote

# initialize repository in notary
notary ${NOTARY_OPTS} init ${MSR_URL}/${NAMESPACE}/${REPO}

# publish locally staged changes
notary ${NOTARY_OPTS} publish ${MSR_URL}/${NAMESPACE}/${REPO}

# rotate snapshot key and change it to server managed
notary ${NOTARY_OPTS} key rotate ${MSR_URL}/${NAMESPACE}/${REPO} snapshot --server-managed

# create delegation for 'targets/releases' role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/releases --all-paths cert.pem

# create delegation for 'targets/${ROLE}' role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/${ROLE} --all-paths cert.pem

# show delegations
notary ${NOTARY_OPTS} delegation list ${MSR_URL}/${NAMESPACE}/${REPO}

# load the key on the client
notary ${NOTARY_OPTS} key import key.pem

# enable DCT
export DOCKER_CONTENT_TRUST=1

# push image
docker push ${MSR_URL}/${NAMESPACE}/${REPO}:<tag>

When adding a delegation, you must also add the delegation for the targets/releases role and it’s recommended that you also add a role to indicate the user or team.

For example, if you add three teams – developer, qa, and devops – you must add them to the targets/releases role because Docker uses the targets/releases role to determine if there is a signature. You can also create a role for each key as shown in this example.

# create delegation for the targets/releases role for each of the keys

notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/releases --all-paths cert.pem

notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO}targets/releases --all-paths ~/developer.pem ~/qa.pem ~/devops.pem

# create delegation to the targets/developer role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/developer --all-paths ~/developer.pem

# create delegation to the targets/qa role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/qa --all-paths ~/qa.pem

# create delegation to the targets/devops role
notary ${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${REPO} targets/devops --all-paths ~/devops.pem

If everyone is pushing to the same tag, you end up with the same hash in targets/developer, targets/qa, and targets/devops, and then whomever signed last also signed the same hash into targets/releases.

If you don’t have a signature on targets/releases, you can’t pull the image. If you create individual roles for each, you get additional data about which signatures are actually in-place based off of the key that was used.

Adding delegations in PowerShell using notary¶

This example shows you how to do the same thing in PowerShell.

# set variables to make this copy and paste
$env:MSR_URL = "msr.example.com"
$env:NAMESPACE = "demo"
$env:REPO = "dcttest"
$env:ROLE = "demo"

# remove data; local and remote (ONLY USED TO RESET CURRENT ENVIRONMENT! WILL REMOVE ALL NOTARY DATA!)
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delete ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} --remote

# initialize repository in notary
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} init ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO}

# publish locally staged changes
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} publish ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO}

# rotate snapshot key and change it to server managed
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} key rotate ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} snapshot --server-managed

# create delegation for 'targets/releases' role
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delegation add -p ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} targets/releases --all-paths cert.pem

# create delegation for 'targets/${ROLE}' role
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delegation add -p ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO} targets/${env:ROLE} --all-paths cert.pem

# show delegations
notary -d ${HOME}\.docker\trust -s https://${env:MSR_URL} delegation list ${env:MSR_URL}/${env:NAMESPACE}/${env:REPO}

# import key
notary -d ${HOME}\.docker\trust key import key.pem

# enabled DCT
$env:DOCKER_CONTENT_TRUST = "1"

Adding delegations using docker trust¶

You can also add delegations using docker trust.

The following commands assume that you already have a client bundle downloaded and extracted to the present working directory.

# login to MSR as the docker trust commands utilize your stored credentials for authentication
$ docker login msr.example.com
Username: admin
Password:

# set the proper environment variables for creating a delegation (only required if you want to not be prompted for passphrases for the DCT process)
$ export DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE="myrootpassphrase"
$ export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="myrepositorypassphrase"


# take your public key and add your user as a signer for the repository
$ docker trust signer add --key cert.pem admin msr.example.com/demo/dcttest

# verify that you see your user now listed as a signer
$ docker trust inspect --pretty msr.example.com/demo/dcttest

# set the proper environment variables to import the private key for
signing (only required if you want to not be prompted for passphrases for
the DCT process)
$ export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="myrepositorypassphrase"

# load your private key into your local storage so that it can
be used by Docker to sign
$ docker trust key load key.pem

# enable DCT on push
$ export DOCKER_CONTENT_TRUST=1

# retag an image to push to MSR
$ docker tag hello-world:latest msr.example.com/demo/dcttest:latest

# push the image and sign
$ docker push msr.example.com/demo/dcttest:latest

# verify that the image tag is now signed
$ docker trust inspect --pretty msr.example.com/demo/dcttest

Use Cases¶

Enabling DCT in Mirantis Container Runtime¶

DCT is controlled by MCR’s configuration file, daemon.json.

MCR verifies that images are signed before starting, which affects the docker run, docker pull, and docker build commmands, but does not affect Swarm services.

The content-trust flag is based around a mode variable instructing MCR whether to enforce signed images, and a trust-pinning variable instructing MCR which sources to trust.

Mode can take three variables:

• Disabled - Verification is not active and the remainder of the content-trust related metadata will be ignored. This is the default value if mode is not specified.

• Permissive - Verification will be performed, but only failures will be logged and remain unenforced. This configuration is intended for testing of changes related to content-trust. The results of the signature verification is displayed in the Mirantis Container Runtime’s daemon logs.

• Enforced - DCT will be enforced and an image that cannot be verified successfully will not be pulled or run.

Here’s an example that sets the flag to enforced.

{
     "content-trust": {
         "mode": "enforced"
     }
 }

Every time the config is changed, the daemon needs to be reloaded.

Official Images¶

The root key for all docker.io/library/* is pinned in MCR itself. Therefore you could limit MCR to only run official docker images.

{
"content-trust": {
   "trust-pinning": {
      "official-library-images": true
   },
   "mode": "enforced"
}
}

User Signed Images¶

Mirantis Container Runtime provides two options for trust-pinning user-signed images.

• Notary Canonical Root key ID (DCT root-key)

  An ID that describes just the root key used to sign a repository (or rather its respective keys).This is the root key on the host that originally signed the repository (for example a personal workstation).

  It can be retrieved from the workstation that signed the repository by running grep -r "root" ~/.docker/trust/private/, or by manually building notary: canonical-info and using the notary info <GUN> command (where GUN stands for Globally Unique Name). This canonical ID should have initiated multiple image repositories, for example mymsr/user1/image1 and mymsr/user1/image2.

• Notary Root key ID (DCT Cert ID)

  An ID that describes the DCT root-key, but it is unique per repository. For example mymsr/user1/image1 and mymsr/usr1/image2 will have unique DCT Cert IDs. You can get the DCT CertID using a $ docker trust inspect command. You could use this method if every root repository was signed with a different host and therefore a different canonical root. Because DCT cert-id is more specific than the root ID, it takes priority or root ID.

Here’s an example daemon.json.

# Retrieving Cert ID
$ docker trust inspect msr.example.com/admin/demo | jq -r '.[].AdministrativeKeys[] | select(.Name=="Root") | .Keys[].ID'

# Retrieving Root ID
$ grep -r "root" ~/.docker/trust/private
/home/example/.docker/trust/private/0b6101527b2ac766702e4b40aa2391805b70e5031c04714c748f914e89014403.key:role: root

# Using a Canonical ID that has signed 2 repos (mymsr/user1/repo1 and mymsr/user1/repo2). Note you can use a Wildcard.

{
"content-trust": {
   "trust-pinning": {
      "root-keys": {
         "mymsr/user1/*": [
         "0b6101527b2ac766702e4b40aa2391805b70e5031c04714c748f914e89014403"
         ]
      }
   },
   "mode": "enforced"
}
}

# Using Cert Ids, by specifying 2 repositories by their DCT root ID. #Example for using this may be different MSRs or maybe because the #repository was initiated on different hosts, therefore having different #canonical IDs.

{
"content-trust": {
   "trust-pinning": {
      "cert-ids": {
         "mymsr/user1/repo1": [
         "9430d6e31e3b3e240957a1b62bbc2d436aafa33726d0fcb50addbf7e2dfa2168"
         ],
         "mymsr/user2/repo1": [
         "544cf09f294860f9d5bc953ad80b386063357fd206b37b541bb2c54166f38d08"
         ]
      }
   },
   "mode": "enforced"
}
}

# Putting it all together

{
"content-trust": {
   "trust-pinning": {
      "official-library-images": true,
      "root-keys": {
         "mymsr/user1/*": [
         "0b6101527b2ac766702e4b40aa2391805b70e5031c04714c748f914e89014403"
         ]
      },
      "cert-ids": {
         "mymsr/user2/repo1": [
         "9430d6e31e3b3e240957a1b62bbc2d436aafa33726d0fcb50addbf7e2dfa2168"
         ],
      }
   },
   "mode": "enforced",
   # Enable offline environments if the engine can't communicate with the
   # registry
   "allow-expired-cached-trust-data": true

}
}

Automation¶

Here’s an example of how to automate delegation. It uses a new feature is available to be able to use NOTARY_AUTH as a base64 encoded username:password.

See Notary client configuration file for more details.

1. Add MSR credentials

   export USERNAME="admin"
   export PASSWORD="docker123"
   

2. Set a flag to create repos

   export CREATE_REPOS=false
   

3. Set environment variables

   export MSR_URL="msr.example.com"
   export NAMESPACE="official"
   export REPO_LIST="alpine centos golang postgres nginx node redis ubuntu"
   export ROLE="demo"
   export USERNAME="username HERE"
   export PASSWORD="password HERE"
   

4. Create notary automation variables

   export NOTARY_ROOT_PASSPHRASE="docker123"
   export NOTARY_TARGETS_PASSPHRASE="docker123"
   export NOTARY_SNAPSHOT_PASSPHRASE="docker123"
   export NOTARY_DELEGATION_PASSPHRASE="docker123"
   export NOTARY_OPTS="-s https://${MSR_URL} -d ${HOME}/.docker/trust"
   

5. Create repos (optional)

   if ${CREATE_REPOS}
   then
     # create org
     curl -sk -X POST --header "Content-Type: application/json" --header "Accept: application/json" -u "${USERNAME}:${PASSWORD}" -d "{
         \"name\": \"official\",
         \"isOrg\": true,
         \"isAdmin\": true,
         \"isActive\": true
       }" "https://${MSR_URL}/enzi/v0/accounts"
   
     # create repos
     for i in ${REPO_LIST}
     do
       curl -sk -X POST --header "Content-Type: application/json" --header "Accept: application/json" -u "${USERNAME}:${PASSWORD}" -d "{
         \"name\": \"${i}\",
         \"shortDescription\": \"Official image for ${i}\",
         \"longDescription\": \"Official image from ${i} from Docker Hub\",
         \"visibility\": \"public\"
       }" "https://${MSR_URL}/api/v0/repositories/official"
     done
   fi
   

6. Write script

   cat > /tmp/notary_expect.exp <<EOL
   #!/usr/bin/env expect -f
   eval spawn notary \$env(NOTARY_PARAMS)
   expect "Enter username: "
   send "\$env(USERNAME)\r"
   expect "Enter password: "
   send "\$env(PASSWORD)\r"
   expect eof
   EOL
   

7. Initialize the repos

   for i in ${REPO_LIST}
   do
      echo -e "\ndelete local and remote data"
      NOTARY_PARAMS="${NOTARY_OPTS} delete ${MSR_URL}/${NAMESPACE}/${i} --remote" expect /tmp/notary_expect.exp
   
      echo -e "\ninitialize repo"
      NOTARY_PARAMS="${NOTARY_OPTS} init ${MSR_URL}/${NAMESPACE}/${i}" expect /tmp/notary_expect.exp
   
      echo -e "\npublish staged changes"
      NOTARY_PARAMS="${NOTARY_OPTS} publish ${MSR_URL}/${NAMESPACE}/${i}" expect /tmp/notary_expect.exp
   
      echo -e "\nrotate snapshot key"
      NOTARY_PARAMS="${NOTARY_OPTS} key rotate ${MSR_URL}/${NAMESPACE}/${i} snapshot --server-managed" expect /tmp/notary_expect.exp
   
      echo -e "\nadd cert to releases role"
      NOTARY_PARAMS="${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${i} targets/releases --all-paths cert.pem" expect /tmp/notary_expect.exp
   
      echo -e "\nadd cert to ${ROLE} role"
      NOTARY_PARAMS="${NOTARY_OPTS} delegation add -p ${MSR_URL}/${NAMESPACE}/${i} targets/${ROLE} --all-paths cert.pem" expect /tmp/notary_expect.exp
   
      echo -e "\nlisting delegations"
      notary ${NOTARY_OPTS} delegation list ${MSR_URL}/${NAMESPACE}/${i}
   done
   

8. Cleanup

   rm /tmp/notary_expect.exp
   

Troubleshooting¶

Here are common errors and how to work around them.

1. Potential malicious behavior error

   Warning: potential malicious behavior - trust data has insufficient signatures for remote repository dtr-beta.demo.dckr.org/official/alpine: valid signatures did not meet threshold

   To fix this error, run notary delete on the client (without the remote flag). For example:

   notary -d ~/.docker/trust delete dtr.fqdn/namespace/repo
   

2. Trusted root error

   Could not validate the path to a trusted root: failed to validate data with current trusted certificates

   You might see this error when attempting to push with a signature after deleting signing data from a repo. The error is raised because of a trusted_certificates directory under tuf that should not be there. 1.12 ignores it, but old versions use it for pinning.

   To fix this error, upgrade to 1.12 or remove the deprecated directory.

3. Failed to validate data error

   {“level”:”debug”,”msg”:”notary.Middleware: Err getting targets: could not rotate trust to a new trusted root: failed to validate data with current trusted certificates”,”time”:”2016-11-09T22:25:18Z”}

   You will see this error when pulling and image after the DCT keys are reset and signed with a new key. MKE continues to store the old key, causing the error.

   To fix it, remove key data using the following example code.

   curl -v -X DELETE -H "Authorization: Bearer $(curl -sk -d '{"username":"username","password":"password"}' https://mke.example.com/auth/login | jq -r .auth_token)" https://mke.example.com/api/trust/namespace/repo
   

4. Missing valid signer error

   Error response from daemon: required at least one valid signer, none found

   This is the error that the user would actually see when trying to deploy a service where:

   1. There isn’t any trust data for the given tag

   2. The user’s delegation didn’t grant them access to the targets/releases role

   3. The trust data doesn’t match any users in MKE

   4. An image being pulled with DCT and if the keys are completely reset and signed with a different key, MKE continues to store that key in etcd. To fix it, remove the key data (Add endpoint to delete pinned notary trust data for a repoCLOSED ):

   To fix this ereror, run the following command, where msr.example.com/demo/docker-demo is the Globally Unique Name (GUN) of your repo.

   docker exec -it -e ETCDCTL_API=3 ucp-kv etcdctl --endpoints https://127.0.0.1:2379 --cert /etc/docker/ssl/cert.pem --key /etc/docker/ssl/key.pem --cacert /etc/docker/ssl/ca.pem del notary/v1/msr.example.com/demo/docker-demo --prefix
   

Related topics¶

• Using Docker Content Trust with a Remote MKE Cluster

• Run only the images you trust