Collect cluster logs

While operating your management, regional, or managed cluster, you may require collecting and inspecting the cluster logs to analyze cluster events or troubleshoot issues.

To collect cluster logs:

1. Verify that the bootstrap directory is updated.

   Select from the following options:

   • For clusters deployed using Container Cloud 2.11.0 or later:

     ./container-cloud bootstrap download --management-kubeconfig <pathToMgmtKubeconfig> \
     --target-dir <pathToBootstrapDirectory>
     

   • For clusters deployed using the Container Cloud release earlier than 2.11.0 or if you deleted the kaas-bootstrap folder, download and run the Container Cloud bootstrap script:

     wget https://binary.mirantis.com/releases/get_container_cloud.sh
     
     chmod 0755 get_container_cloud.sh
     
     ./get_container_cloud.sh
     

2. Obtain kubeconfig of the required cluster. The management or regional cluster kubeconfig files are created during the last stage of the management or regional cluster bootstrap. To obtain a managed cluster kubeconfig, see Connect to a Mirantis Container Cloud cluster.

3. Obtain the private SSH key of the required cluster:

   • For a managed cluster, this is an SSH key added in the Container Cloud web UI before the managed cluster creation.

   • For a management or regional cluster, ssh_key is created during bootstrap of the corresponding cluster in the same directory as the bootstrap script.

     Note

     If the initial version of your Container Cloud management cluster was earlier than 2.6.0, ssh_key is named openstack_tmp and is located at ~/.ssh/.

4. Depending on the cluster type that you require logs from, run the corresponding command:

   • For a management cluster:

     ./container-cloud collect logs --management-kubeconfig <pathToMgmtClusterKubeconfig> \
     --key-file <pathToMgmtClusterPrivateSshKey> \
     --cluster-name <clusterName> --cluster-namespace <clusterProject>
     

   • For a regional cluster:

     ./container-cloud collect logs --management-kubeconfig <pathToMgmtClusterKubeconfig> \
     --key-file <pathToRegionalClusterSshKey> --kubeconfig <pathToRegionalClusterKubeconfig> \
     --cluster-name <clusterName> --cluster-namespace <clusterProject>
     

   • For a managed cluster:

     ./container-cloud collect logs --management-kubeconfig <pathToMgmtClusterKubeconfig> \
     --key-file <pathToManagedClusterSshKey> --kubeconfig <pathToManagedClusterKubeconfig> \
     --cluster-name <clusterName> --cluster-namespace <clusterProject>
     

   Substitute the parameters enclosed in angle brackets with the corresponding values of your cluster.

   Optional flags:

   • --output-dir

     Directory path to save logs. The default value is logs/. For example, logs/<clusterName>/events.log.

   • --extended

     Output the extended version of logs that contains system and MKE logs, logs from LCM Ansible and LCM Agent along with cluster events and Kubernetes resources description and logs.

     Without the --extended flag, the basic version of logs is collected, which is sufficient for most use cases. The basic version of logs contains all events, Kubernetes custom resources, and logs from all Container Cloud components. This version does not require passing --key-file.

   For the logs structure, see Collect the bootstrap logs.

5. If you require logs of a cluster update, inspect the following folders on the control plane nodes:

   • /objects/namespaced/<namespaceName>/core/pods/lcm-lcm-controller- <controllerID>/ for the lcm-controller logs.

   • /objects/namespaced/<namespaceName>/core/pods/<cloudProviderName-ID>/ for logs of the cloud provider controller. For example, vsphere-provider-5b96fb4fd6-bhl7g.

   • /system/mke/<controllerMachineName>/ (or /system/<controllerMachineName>/mke/) for the MKE support dump. The dsinfo/dsinfo.txt file contains Docker and system information of the MKE configuration set before and after update.

   • events.log for cluster events logs.

6. Technology Preview. For bare metal clusters, assess the Ironic pod logs:

   • Extract the content of the 'message' fields from every log message:

     kubectl -n kaas logs <ironicPodName> -c syslog | jq -rM '.message'
     

   • Extract the content of the 'message' fields from the ironic_conductor source log messages:

     kubectl -n kaas logs <ironicPodName> -c syslog | jq -rM 'select(.source == "ironic_conductor") | .message'
     

   The syslog container collects logs generated by Ansible during the node deployment and cleanup and outputs them in the JSON format.

7. Compress the collected log files and send the archive to the Mirantis support team.