Troubleshoot using logs

You can troubleshoot your MKE cluster by reviewing the logs of the individual MKE components using the MKE web UI, CLI, and support dump. You must have administrator privileges to view information about MKE system containers.


To review the logs using the MKE web UI:

  1. Log in to the MKE web UI as an administrator.

  2. In the left-side navigation panel, navigate to Shared Resources > Containers. By default, the system containers are hidden.

  3. Click the gear icon and select Show system resources.

  4. Click the required container to view details, which include configurations and logs.


To review the logs using the CLI:

  1. Download and configure the client bundle.

    Using the Docker CLI requires that you authenticate using client certificates. Client certificate bundles generated for users without administrator privileges do not permit viewing MKE system container logs.

  2. Review the logs of MKE system containers. Use the -a flag to display system containers, as they are not displayed by default.

    docker ps -a
    

    Example output:

    CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS                     PORTS                                                                             NAMES
    8b77cfa87889        mirantis/ucp-agent:latest             "/bin/ucp-agent re..."   3 hours ago         Exited (0) 3 hours ago                                                                                       ucp-reconcile
    b844cf76a7a5        mirantis/ucp-agent:latest             "/bin/ucp-agent agent"   3 hours ago         Up 3 hours                 2376/tcp                                                                          ucp-agent.tahzo3m4xjwhtsn6l3n8oc2bf.xx2hf6dg4zrphgvy2eohtpns9
    de5b45871acb        mirantis/ucp-controller:latest        "/bin/controller s..."   3 hours ago         Up 3 hours (unhealthy)     0.0.0.0:443->8080/tcp                                                             ucp-controller
    ...
    
  3. Optional. Review the log of a particular MKE container by using the docker logs <mke container ID> command. For example, the following command produces the log for the ucp-controller container listed in the previous step:

    docker logs de5b45871acb
    

    Example output:

    {"level":"info","license_key":"PUagrRqOXhMH02UgxWYiKtg0kErLY8oLZf1GO4Pw8M6B","msg":"/v1.22/containers/ucp/ucp-controller/json",
    "remote_addr":"192.168.10.1:59546","tags":["api","v1.22","get"],"time":"2016-04-25T23:49:27Z","type":"api","username":"dave.lauper"}
    {"level":"info","license_key":"PUagrRqOXhMH02UgxWYiKtg0kErLY8oLZf1GO4Pw8M6B","msg":"/v1.22/containers/ucp/ucp-controller/logs",
    "remote_addr":"192.168.10.1:59546","tags":["api","v1.22","get"],"time":"2016-04-25T23:49:27Z","type":"api","username":"dave.lauper"}
    

To review the logs using a support dump:

Using the logs contained in a support dump allows you to troubleshoot problems that existed prior to changing your MKE configuration. Do not alter your MKE configuration until after you have performed the following steps.

  1. Log in to the MKE web UI.

  2. In the left-side navigation panel, navigate to <username> > Admin Settings > Log & Audit Logs

  3. Select DEBUG and click Save.

    Increasing the MKE log level to DEBUG produces more descriptive logs, making it easier to understand the status of the MKE cluster.

    Note

    Changing the MKE log level restarts all MKE system components and introduces a small amount of downtime to MKE. Your applications will not be affected by this downtime.

  4. support-dump.

Each of the following container types reports a different variety of problems in its logs:

  • Review the ucp-reconcile container logs for problems that occur after a node was added or removed.

    Note

    It is normal for the ucp-reconcile container to be stopped. This container starts only when the ucp-agent detects that a node needs to transition to a different state. The ucp-reconcile container is responsible for creating and removing containers, issuing certificates, and pulling missing images.

  • Review the ucp-controller container logs for problems that occur in the normal state of the system.

  • Review the ucp-auth-api and ucp-auth-store container logs for problems that occur when you are able to visit the MKE web UI but unable to log in.