Monitor MSR

Mirantis Secure Registry is a Dockerized application. To monitor it, you can use the same tools and techniques you’re already using to monitor other containerized applications running on your cluster. One way to monitor MSR is using the monitoring capabilities of Docker Universal Control Plane.

In your browser, log in to Mirantis Kubernetes Engine (MKE), and navigate to the Stacks page. If you have MSR set up for high-availability, then all the MSR replicas are displayed.

To check the containers for the MSR replica, click the replica you want to inspect, click Inspect Resource, and choose Containers.

Now you can drill into each MSR container to see its logs and find the root cause of the problem.

Health check endpoints

MSR also exposes several endpoints you can use to assess if a MSR replica is healthy or not:

  • /_ping: Checks if the MSR replica is healthy, and returns a simple json response. This is useful for load balancing or other automated health check tasks.

  • /nginx_status: Returns the number of connections being handled by the NGINX front-end used by MSR.

  • /api/v0/meta/cluster_status: Returns extensive information about all MSR replicas.

Cluster status

The /api/v0/meta/cluster_status endpoint requires administrator credentials, and returns a JSON object for the entire cluster as observed by the replica being queried. You can authenticate your requests using HTTP basic auth.

curl -ksL -u <user>:<pass> https://<msr-domain>/api/v0/meta/cluster_status
{
  "current_issues": [
   {
    "critical": false,
    "description": "... some replicas are not ready. The following servers are
                    not reachable: dtr_rethinkdb_f2277ad178f7",
  }],
  "replica_health": {
    "f2277ad178f7": "OK",
    "f3712d9c419a": "OK",
    "f58cf364e3df": "OK"
  },
}

You can find health status on the current_issues and replica_health arrays. If this endpoint doesn’t provide meaningful information when trying to troubleshoot, try troubleshooting using logs.

Check notary audit logs

Docker Content Trust (DCT) keeps audit logs of changes made to trusted repositories. Every time you push a signed image to a repository, or delete trust data for a repository, DCT logs that information.

These logs are only available from the MSR API.

Get an authentication token

To access the audit logs you need to authenticate your requests using an authentication token. You can get an authentication token for all repositories, or one that is specific to a single repository.

curl --insecure --silent \
  --user <user>:<password> \
  "https://<dtr-url>/auth/token?realm=dtr&service=dtr&scope=registry:catalog:*"
curl --insecure --silent \
  --user <user>:<password> \
  "https://<dtr-url>/auth/token?realm=dtr&service=dtr&scope=repository:<dtr-url>/<repository>:pull"

MSR returns a JSON file with a token, even when the user doesn’t have access to the repository to which they requested the authentication token. This token doesn’t grant access to MSR repositories.

The JSON file returned has the following structure:

{
  "token": "<token>",
  "access_token": "<token>",
  "expires_in": "<expiration in seconds>",
  "issued_at": "<time>"
}

Changefeed API

Once you have an authentication token you can use the following endpoints to get audit logs:

URL

Description

Authorization

GET /v2/_trust/changefeed

Get audit logs for all repositories.

Global scope token

GET /v2/<msr-url>/<repository>/_trust/changefeed

Get audit logs for a specific repository.

Repositorhy-specific token

Both endpoints have the following query string parameters:

Field name

Required

Type

Description

change_id

Yes

String

A non-inclusive starting change ID from which to start returning results. This will typically be the first or last change ID from the previous page of records requested, depending on which direction your are paging in.

The value 0 indicates records should be returned starting from the beginning of time.

The value 1 indicates records should be returned starting from the most recent record. If 1 is provided, the implementation will also assume the records value is meant to be negative, regardless of the given sign.

records

Yes

String integer

The number of records to return. A negative value indicates the number of records preceding the change_id should be returned. Records are always returned sorted from oldest to newest.

The response is a JSON like:

{
  "count": 1,
  "records": [
    {
      "ID": "0a60ec31-d2aa-4565-9b74-4171a5083bef",
      "CreatedAt": "2017-11-06T18:45:58.428Z",
      "GUN": "msr.example.org/library/wordpress",
      "Version": 1,
      "SHA256": "a4ffcae03710ae61f6d15d20ed5e3f3a6a91ebfd2a4ba7f31fc6308ec6cc3e3d",
      "Category": "update"
    }
  ]
}

Below is the description for each of the fields in the response:

Field name

Description

count

The number of records returned.

ID

The ID of the change record. Should be used in the change_id field of requests to provide a non-exclusive starting index. It should be treated as an opaque value that is guaranteed to be unique within an instance of notary.

CreatedAt

The time the change happened.

GUN

The MSR repository that was changed.

Version

The version that the repository was updated to. This increments every time there’s a change to the trust repository.

This is always 0 for events representing trusted data being removed from the repository.

SHA256

The checksum of the timestamp being updated to. This can be used with the existing notary APIs to request said timestamp.

This is always an empty string for events representing trusted data being removed from the repository

Category

The kind of change that was made to the trusted repository. Can be update, or deletion.

The results only include audit logs for events that happened more than 60 seconds ago, and are sorted from oldest to newest.

Even though the authentication API always returns a token, the changefeed API validates if the user has access to see the audit logs or not:

  • If the user is an admin they can see the audit logs for any repositories,

  • All other users can only see audit logs for repositories they have read access.