Troubleshoot using logs¶

You can troubleshoot your MKE cluster by using the MKE web UI, the ClI, and the support bundle to review the logs of the individual MKE components. You must have administrator privileges to view information about MKE system containers.

Review logs using the MKE web UI¶

Log in to the MKE web UI as an administrator.
In the left-side navigation panel, navigate to Shared Resources > Containers. By default, the system containers are hidden.
Click the gear icon and select Show system resources.
Click the required container to view details, which include configurations and logs.

Review logs using the CLI¶

Download and configure the client bundle.

Using the Docker CLI requires that you authenticate using client certificates. Client certificate bundles generated for users without administrator privileges do not permit viewing MKE system container logs.

Review the logs of MKE system containers. Use the -a flag to display system containers, as they are not displayed by default.

docker ps -a

Example output:

CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS                     PORTS                                                                             NAMES
8b77cfa87889        mirantis/ucp-agent:latest             "/bin/ucp-agent re..."   3 hours ago         Exited (0) 3 hours ago                                                                                       ucp-reconcile
b844cf76a7a5        mirantis/ucp-agent:latest             "/bin/ucp-agent agent"   3 hours ago         Up 3 hours                 2376/tcp                                                                          ucp-agent.tahzo3m4xjwhtsn6l3n8oc2bf.xx2hf6dg4zrphgvy2eohtpns9
de5b45871acb        mirantis/ucp-controller:latest        "/bin/controller s..."   3 hours ago         Up 3 hours (unhealthy)     0.0.0.0:443->8080/tcp                                                             ucp-controller
...

Optional. Review the log of a particular MKE container by using the docker logs <mke container ID> command. For example, the following command produces the log for the ucp-controller container listed in the previous step:

docker logs de5b45871acb

Example output:

{"level":"info","license_key":"PUagrRqOXhMH02UgxWYiKtg0kErLY8oLZf1GO4Pw8M6B","msg":"/v1.22/containers/ucp/ucp-controller/json",
"remote_addr":"192.168.10.1:59546","tags":["api","v1.22","get"],"time":"2016-04-25T23:49:27Z","type":"api","username":"dave.lauper"}
{"level":"info","license_key":"PUagrRqOXhMH02UgxWYiKtg0kErLY8oLZf1GO4Pw8M6B","msg":"/v1.22/containers/ucp/ucp-controller/logs",
"remote_addr":"192.168.10.1:59546","tags":["api","v1.22","get"],"time":"2016-04-25T23:49:27Z","type":"api","username":"dave.lauper"}

Review logs using a support bundle¶

With the logs contained in a support bundle you can troubleshoot problems that existed before you changed your MKE configuration. Do not alter your MKE configuration until after you have performed the following steps.

Log in to the MKE web UI.
In the left-side navigation panel, navigate to <username> > Admin Settings > Log & Audit Logs
Select DEBUG and click Save.

Increasing the MKE log level to DEBUG produces more descriptive logs, making it easier to understand the status of the MKE cluster.

Note

Changing the MKE log level restarts all MKE system components and introduces a small amount of downtime to MKE. Your applications will not be affected by this downtime.
support-dump.

Each of the following container types reports a different variety of problems in its logs:

Review the ucp-reconcile container logs for problems that occur after a node was added or removed.

Note

It is normal for the ucp-reconcile container to be stopped. This container starts only when the ucp-agent detects that a node needs to transition to a different state. The ucp-reconcile container is responsible for creating and removing containers, issuing certificates, and pulling missing images.
Review the ucp-controller container logs for problems that occur in the normal state of the system.
Review the ucp-auth-api and ucp-auth-store container logs for problems that occur when you are able to visit the MKE web UI but unable to log in.

Review logs using the API¶

Store the IP address for use in the shell:
```
IP=<ip-address>
```

Obtain a temporary access token:

curl -k -X POST -H 'Content-Type: application/json' https://$IP/auth/login --data-binary '
{
  "username": "<username>",
  "password": "<password>"
}
'

Example output:

{"auth_token":"88d790ab-5cc0-4284-b3c6-986272af50b6"}

Store the temporary access token for use in the shell:
```
AUTHTOKEN="88d790ab-5cc0-4284-b3c6-986272af50b6"
```

Determine which containers are present:

curl -k -X GET "https://$IP/containers/json?all=true&size=false" -H  "accept: application/json" -H  "Authorization: Bearer ${AUTHTOKEN}"

Truncated first line of example output:

{"Id":"2cebeb898636ce519ec68fadbad4abe499f2fdebb057eb534bb64ad5bbf7925f", ...}

Store the container ID for use in the shell:

ID=2cebeb898636ce519ec68fadbad4abe499f2fdebb057eb534bb64ad5bbf7925f

Obtain log files associated with the container ID:

curl -k -X GET "https://$IP/containers/$ID/logs?follow=false&stdout=true&stderr=true&since=0&until=0&timestamps=false&tail=all" \
-H "accept: application/json" -H  "Authorization: Bearer ${AUTHTOKEN}" --output output.txt

View log content:
```
cat output.txt
```