Troubleshoot your MSR Swarm deployment¶
The commands herein allow you to diagnose and resolve common issues you may encounter in deploying MSR on a Swarm cluster.
To identify a failed service on your cluster:
List the services in your MSR stack and subsequently identify any that are not running.
docker stack services msr
Example output:
ID NAME MODE REPLICAS IMAGE PORTS
k8taishq5xxk msr_msr-api-server replicated 3/3 registry.mirantis.com/msr/msr-api:<release number>
fk344mcex0gp msr_msr-enzi-api replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
p75o0wug72ck msr_msr-enzi-worker replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
bnulom7u88fd msr_msr-garant replicated 3/3 registry.mirantis.com/msr/msr-garant:<release number>
p14k98kl9tt6 msr_msr-initialize replicated 0/1 registry.mirantis.com/msr/msr-api:<release number>
k5qsenngjxc4 msr_msr-jobrunner replicated 3/3 registry.mirantis.com/msr/msr-jobrunner:<release number>
qv3cdf30ebbb msr_msr-nginx replicated 3/3 registry.mirantis.com/msr/msr-nginx:<release number> *:443->443/tcp, *:8080->8080/tcp
eroxakg061ns msr_msr-notary-server replicated 3/3 registry.mirantis.com/msr/msr-notary-server:<release number>
8osnskkpvv9d msr_msr-notary-signer replicated 3/3 registry.mirantis.com/msr/msr-notary-signer:<release number>
v9q1e6nnzutq msr_msr-registry replicated 0/3 registry.mirantis.com/msr/msr-registry:<release number>
o32erkkz8tjo msr_msr-rethinkdb replicated 3/3 mirantis/rethinkdb:2.3.7-mirantis-41-a02bade
To obtain detailed information for a service that is not running:
docker service ps msr_msr-registry --no-trunc
Example output:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
7o8rjdjydwfqnz0qhekz46tq5 msr_msr-registry.1 registry.mirantis.com/msr/msr-registry:<release number>@sha256:a4d3a083da310dff374c37850e1e8de81ad9150b770683b1529cabf508ae8f07 6e1b4b0f0dcc Ready Ready 1 second ago
lickekmwnp6d2ot558ohh2cnj \_ msr_msr-registry.1 registry.mirantis.com/msr/msr-registry:<release number>@sha256:a4d3a083da310dff374c37850e1e8de81ad9150b770683b1529cabf508ae8f07 aed603d27071 Shutdown Failed 1 second ago "starting container failed: error while mounting volume '/var/lib/docker/volumes/msr_msr-storage/_data': failed to mount local volume: mount :/:/var/lib/docker/volumes/msr_msr-storage/_data, data: addr=172.17.0.10,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport: connection refused"
To review all of the services that are running on the cluster:
docker service ls
Example output:
ID NAME MODE REPLICAS IMAGE PORTS
sr1ivj8c0iyh msr_msr-api-server replicated 3/3 registry.mirantis.com/msr/msr-api:<release number>
ks7r7nctqaon msr_msr-enzi-api replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
rj7z7iojd54g msr_msr-enzi-worker replicated 3/3 registry.mirantis.com/msr/enzi:1.0.85
n7mufyqsl8n3 msr_msr-garant replicated 3/3 registry.mirantis.com/msr/msr-garant:<release number>
s0p4vmxopdbt msr_msr-initialize replicated 0/1 registry.mirantis.com/msr/msr-api:<release number>
llvu69o504ks msr_msr-jobrunner replicated 3/3 registry.mirantis.com/msr/msr-jobrunner:<release number>
kycj3hoqd74s msr_msr-nginx replicated 3/3 registry.mirantis.com/msr/msr-nginx:<release number> *:443->443/tcp, *:8080->8080/tcp
jsxdq6j25r7h msr_msr-notary-server replicated 3/3 registry.mirantis.com/msr/msr-notary-server:<release number>
3zrjhpe2rb4i msr_msr-notary-signer replicated 3/3 registry.mirantis.com/msr/msr-notary-signer:<release number>
znz4ioqyegkt msr_msr-registry replicated 3/3 registry.mirantis.com/msr/msr-registry:<release number>
lm47q08a7t9i msr_msr-rethinkdb replicated 3/3 mirantis/rethinkdb:2.3.7-mirantis-41-a02bade
To obtain the service logs:
docker service logs msr_msr-api-server
Example output:
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Generating an authenticator for eNZi client","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Attempting to create or update MSR's Service registration with the eNZi server","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Updated service \"Mirantis Secure Registry\"","time":"2023-06-27T23:01:47Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Obtaining eNZi service registration","time":"2023-06-27T23:01:48Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"error","msg":"failed to obtain repository counts: rethinkdb: Cannot reduce over an empty stream. in:\nr.DB(\"dtr2\").Table(\"repositories\").Group(\"visibility\").Count().Ungroup().Map(func(var_2 r.Term) r.Term { return r.Object(var_2.Field(\"group\"), var_2.Field(\"reduction\")) }).Reduce(func(var_3, var_4 r.Term) r.Term { return var_3.Merge(var_4) })","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"Starting temporary CVE file cleanup within \"/storage/scan_update/\" directory","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"error":"open /storage/scan_update/: no such file or directory","level":"error","msg":"Could not delete all tmp files","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"level":"info","msg":"No files to remove","time":"2023-06-27T23:01:49Z"}
msr_msr-api-server.3.iippai90ljtr@c1138be288cc | {"address":":443","level":"info","msg":"Admin server about to listen for connections","time":"2023-06-27T23:01:49Z"}
To create a shell to examine the contents of a container:
SSH into the host that is running the container to which you want to connect.
Obtain the required container ID:
CONTAINER_ID=docker ps --filter="name=<container-name>*"
Run a shell within the required container:
docker exec -it $CONTAINER_ID sh