With the upcoming end-of-life (EOL) of MSR 3.1.x, Mirantis encourages you to upgrade to the latest supported version to ensure continued support and security updates.

Troubleshoot your MSR Swarm deployment

The commands herein allow you to diagnose and resolve common issues you may encounter in deploying MSR on a Swarm cluster.

To identify a failed service on your cluster:

List the services in your MSR stack and subsequently identify any that are not running.

docker stack services msr

Example output:

ID             NAME                    MODE         REPLICAS   IMAGE                                                   PORTS
k8taishq5xxk   msr_msr-api-server      replicated   3/3        registry.mirantis.com/msr/msr-api:<release number>
fk344mcex0gp   msr_msr-enzi-api        replicated   3/3        registry.mirantis.com/msr/enzi:1.0.85
p75o0wug72ck   msr_msr-enzi-worker     replicated   3/3        registry.mirantis.com/msr/enzi:1.0.85
bnulom7u88fd   msr_msr-garant          replicated   3/3        registry.mirantis.com/msr/msr-garant:<release number>
p14k98kl9tt6   msr_msr-initialize      replicated   0/1        registry.mirantis.com/msr/msr-api:<release number>
k5qsenngjxc4   msr_msr-jobrunner       replicated   3/3        registry.mirantis.com/msr/msr-jobrunner:<release number>
qv3cdf30ebbb   msr_msr-nginx           replicated   3/3        registry.mirantis.com/msr/msr-nginx:<release number>            *:443->443/tcp, *:8080->8080/tcp
eroxakg061ns   msr_msr-notary-server   replicated   3/3        registry.mirantis.com/msr/msr-notary-server:<release number>
8osnskkpvv9d   msr_msr-notary-signer   replicated   3/3        registry.mirantis.com/msr/msr-notary-signer:<release number>
v9q1e6nnzutq   msr_msr-registry        replicated   0/3        registry.mirantis.com/msr/msr-registry:<release number>
o32erkkz8tjo   msr_msr-rethinkdb       replicated   3/3        mirantis/rethinkdb:2.3.7-mirantis-41-a02bade

To obtain detailed information for a service that is not running:

docker service ps msr_msr-registry --no-trunc

Example output:

ID                          NAME                     IMAGE                                                                                                                             NODE           DESIRED STATE   CURRENT STATE           ERROR                                                                                                                                                                                                                                                                                                                    PORTS
7o8rjdjydwfqnz0qhekz46tq5   msr_msr-registry.1       registry.mirantis.com/msr/msr-registry:<release number>@sha256:a4d3a083da310dff374c37850e1e8de81ad9150b770683b1529cabf508ae8f07   6e1b4b0f0dcc   Ready           Ready 1 second ago
lickekmwnp6d2ot558ohh2cnj    \_ msr_msr-registry.1   registry.mirantis.com/msr/msr-registry:<release number>@sha256:a4d3a083da310dff374c37850e1e8de81ad9150b770683b1529cabf508ae8f07   aed603d27071   Shutdown        Failed 1 second ago     "starting container failed: error while mounting volume '/var/lib/docker/volumes/msr_msr-storage/_data': failed to mount local volume: mount :/:/var/lib/docker/volumes/msr_msr-storage/_data, data: addr=172.17.0.10,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport: connection refused"

To review all of the services that are running on the cluster:

docker service ls

Example output:

ID             NAME                    MODE         REPLICAS   IMAGE                                                   PORTS
sr1ivj8c0iyh   msr_msr-api-server      replicated   3/3        registry.mirantis.com/msr/msr-api:<release number>
ks7r7nctqaon   msr_msr-enzi-api        replicated   3/3        registry.mirantis.com/msr/enzi:1.0.85
rj7z7iojd54g   msr_msr-enzi-worker     replicated   3/3        registry.mirantis.com/msr/enzi:1.0.85
n7mufyqsl8n3   msr_msr-garant          replicated   3/3        registry.mirantis.com/msr/msr-garant:<release number>
s0p4vmxopdbt   msr_msr-initialize      replicated   0/1        registry.mirantis.com/msr/msr-api:<release number>
llvu69o504ks   msr_msr-jobrunner       replicated   3/3        registry.mirantis.com/msr/msr-jobrunner:<release number>
kycj3hoqd74s   msr_msr-nginx           replicated   3/3        registry.mirantis.com/msr/msr-nginx:<release number>           *:443->443/tcp, *:8080->8080/tcp
jsxdq6j25r7h   msr_msr-notary-server   replicated   3/3        registry.mirantis.com/msr/msr-notary-server:<release number>
3zrjhpe2rb4i   msr_msr-notary-signer   replicated   3/3        registry.mirantis.com/msr/msr-notary-signer:<release number>
znz4ioqyegkt   msr_msr-registry        replicated   3/3        registry.mirantis.com/msr/msr-registry:<release number>
lm47q08a7t9i   msr_msr-rethinkdb       replicated   3/3        mirantis/rethinkdb:2.3.7-mirantis-41-a02bade

To obtain the service logs:

docker service logs msr_msr-api-server

Example output:

msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"info","msg":"Generating an authenticator for eNZi client","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"info","msg":"Attempting to create or update MSR's Service registration with the eNZi server","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"info","msg":"Updated service \"Mirantis Secure Registry\"","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"info","msg":"Obtaining eNZi service registration","time":"2023-06-27T23:01:48Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"error","msg":"failed to obtain repository counts: rethinkdb: Cannot reduce over an empty stream. in:\nr.DB(\"dtr2\").Table(\"repositories\").Group(\"visibility\").Count().Ungroup().Map(func(var_2 r.Term) r.Term { return r.Object(var_2.Field(\"group\"), var_2.Field(\"reduction\")) }).Reduce(func(var_3, var_4 r.Term) r.Term { return var_3.Merge(var_4) })","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"info","msg":"Starting temporary CVE file cleanup within \"/storage/scan_update/\" directory","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"error":"open /storage/scan_update/: no such file or directory","level":"error","msg":"Could not delete all tmp files","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"level":"info","msg":"No files to remove","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc    | {"address":":443","level":"info","msg":"Admin server about to listen for connections","time":"2023-06-27T23:01:49Z"}

To create a shell to examine the contents of a container:

  1. SSH into the host that is running the container to which you want to connect.

  2. Obtain the required container ID:

    CONTAINER_ID=docker ps --filter="name=<container-name>*"
    
  3. Run a shell within the required container:

    docker exec -it $CONTAINER_ID sh