With the upcoming end-of-life (EOL) of MSR 3.1.x, Mirantis encourages you to upgrade to the latest supported version to ensure continued support and security updates.
Troubleshoot your MSR Swarm deployment¶
The commands herein allow you to diagnose and resolve common issues you may encounter in deploying MSR on a Swarm cluster.
To identify a failed service on your cluster:
List the services in your MSR stack and subsequently identify any that are not running.
docker stack services msr
Example output:
ID NAME MODE REPLICAS IMAGE PORTS
k8taishq5xxk msr_msr-api-server replicated 3/3 registry.mirantis.com/msr/msr-api:<release number>
fk344mcex0gp msr_msr-enzi-api replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
p75o0wug72ck msr_msr-enzi-worker replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
bnulom7u88fd msr_msr-garant replicated 3/3 registry.mirantis.com/msr/msr-garant:<release number>
p14k98kl9tt6 msr_msr-initialize replicated 0/1 registry.mirantis.com/msr/msr-api:<release number>
k5qsenngjxc4 msr_msr-jobrunner replicated 3/3 registry.mirantis.com/msr/msr-jobrunner:<release number>
qv3cdf30ebbb msr_msr-nginx replicated 3/3 registry.mirantis.com/msr/msr-nginx:<release number> *:443->443/tcp, *:8080->8080/tcp
eroxakg061ns msr_msr-notary-server replicated 3/3 registry.mirantis.com/msr/msr-notary-server:<release number>
8osnskkpvv9d msr_msr-notary-signer replicated 3/3 registry.mirantis.com/msr/msr-notary-signer:<release number>
v9q1e6nnzutq msr_msr-registry replicated 0/3 registry.mirantis.com/msr/msr-registry:<release number>
o32erkkz8tjo msr_msr-rethinkdb replicated 3/3 mirantis/rethinkdb:2.3.7-mirantis-41-a02bade
To obtain detailed information for a service that is not running:
docker service ps msr_msr-registry --no-trunc
Example output:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
7o8rjdjydwfqnz0qhekz46tq5 msr_msr-registry.1 registry.mirantis.com/msr/msr-registry:<release number>@sha256:a4d3a083da310dff374c37850e1e8de81ad9150b770683b1529cabf508ae8f07 6e1b4b0f0dcc Ready Ready 1 second ago
lickekmwnp6d2ot558ohh2cnj \_ msr_msr-registry.1 registry.mirantis.com/msr/msr-registry:<release number>@sha256:a4d3a083da310dff374c37850e1e8de81ad9150b770683b1529cabf508ae8f07 aed603d27071 Shutdown Failed 1 second ago "starting container failed: error while mounting volume '/var/lib/docker/volumes/msr_msr-storage/_data': failed to mount local volume: mount :/:/var/lib/docker/volumes/msr_msr-storage/_data, data: addr=172.17.0.10,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport: connection refused"
To review all of the services that are running on the cluster:
docker service ls
Example output:
ID NAME MODE REPLICAS IMAGE PORTS
sr1ivj8c0iyh msr_msr-api-server replicated 3/3 registry.mirantis.com/msr/msr-api:<release number>
ks7r7nctqaon msr_msr-enzi-api replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
rj7z7iojd54g msr_msr-enzi-worker replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
n7mufyqsl8n3 msr_msr-garant replicated 3/3 registry.mirantis.com/msr/msr-garant:<release number>
s0p4vmxopdbt msr_msr-initialize replicated 0/1 registry.mirantis.com/msr/msr-api:<release number>
llvu69o504ks msr_msr-jobrunner replicated 3/3 registry.mirantis.com/msr/msr-jobrunner:<release number>
kycj3hoqd74s msr_msr-nginx replicated 3/3 registry.mirantis.com/msr/msr-nginx:<release number> *:443->443/tcp, *:8080->8080/tcp
jsxdq6j25r7h msr_msr-notary-server replicated 3/3 registry.mirantis.com/msr/msr-notary-server:<release number>
3zrjhpe2rb4i msr_msr-notary-signer replicated 3/3 registry.mirantis.com/msr/msr-notary-signer:<release number>
znz4ioqyegkt msr_msr-registry replicated 3/3 registry.mirantis.com/msr/msr-registry:<release number>
lm47q08a7t9i msr_msr-rethinkdb replicated 3/3 mirantis/rethinkdb:2.3.7-mirantis-41-a02bade
To obtain the service logs:
docker service logs msr_msr-api-server
Example output:
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Generating an authenticator for eNZi client","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Attempting to create or update MSR's Service registration with the eNZi server","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Updated service \"Mirantis Secure Registry\"","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Obtaining eNZi service registration","time":"2023-06-27T23:01:48Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"error","msg":"failed to obtain repository counts: rethinkdb: Cannot reduce over an empty stream. in:\nr.DB(\"dtr2\").Table(\"repositories\").Group(\"visibility\").Count().Ungroup().Map(func(var_2 r.Term) r.Term { return r.Object(var_2.Field(\"group\"), var_2.Field(\"reduction\")) }).Reduce(func(var_3, var_4 r.Term) r.Term { return var_3.Merge(var_4) })","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Starting temporary CVE file cleanup within \"/storage/scan_update/\" directory","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"error":"open /storage/scan_update/: no such file or directory","level":"error","msg":"Could not delete all tmp files","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"No files to remove","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"address":":443","level":"info","msg":"Admin server about to listen for connections","time":"2023-06-27T23:01:49Z"}
To create a shell to examine the contents of a container:
SSH into the host that is running the container to which you want to connect.
Obtain the required container ID:
CONTAINER_ID=docker ps --filter="name=<container-name>*"
Run a shell within the required container:
docker exec -it $CONTAINER_ID sh