Datadog Monitoring Implementation Guide for Docker Enterprise

Datadog Monitoring Implementation Guide for Docker Enterprise

Overview

MCR, MKE, and MSR Solution Guides enable you to integrate our container platform with popular 3rd party ecosystem solutions for networking, load balancing, storage, logging and monitoring, access management, and more. This Solution Guide explains how to deploy Datadog to monitor the performance of Docker containers in an MKE cluster environment.

MCR includes Cluster, a tool for lifecycle management of Docker clusters. With Cluster, you use a YAML file to configure your provider’s resources. Then, with a single command, you provision and install all the resources from your configuration. The cluster in this Solution Guide was created using Cluster on Azure Cloud.

Note

Mirantis does not provide the product support for Datadog. Please contact the vendor-approved support services if you have any questions or problems with Datadog.

Datadog Overview

Datadog is an SaaS-based data analytics monitoring service for cloud-scale applications, bringing together data from servers, databases, tools, and services to present a unified view of an entire stack.

Datadog’s main features are:

  • Ability for developers and operations to see the full infrastructure on a single pane of glass:

    • Clouds

    • Servers

    • Applications

    • Services

    • Metrics

    • and more

  • Real-time, customizable, interactive dashboards

  • Full-text search capabilities for metrics and events

  • Over 400 integrations which include various clouds, enterprise applications, and developer software tools out of the box

  • API to allow integrating and monitoring unique infrastructures

Datadog provides an agent for all of the major Linux distributions and Windows, which needs to be downloaded and installed on each host that you wish to monitor. The agent collects the events and metrics and then forwards them to the Datadog SaaS server for processing, analysis, and viewing. Datadog also provides their Linux agent as a Certified Docker Image in Docker Hub.

MKE Cluster

This implementation guide was developed and tested on:

  • Mirantis Container Runtime 19.03.2, MKE 3.2.1 and MSR 2.7.1, and DataDog Agent 6.14.1

The MKE Cluster environment has 6 nodes: 1 manager, and 5 workers (for more detail, refer to Docker Enterprise Best Practices and Design Considerations).

To discover your infrastructure and collect the events and metrics, the Datadog agent needs to be deployed on all Docker Nodes that you wish to monitor.

MKE Configuration

Datadog SaaS Account

A Datadog SaaS Account is required. You can register for a free 14-day Datadog Account on datadoghq.com. Click on the GET STARTED FREE button.

Get Started for Free

Datadog API Key

The Datadog agent requires a Datadog API Key in order for it to securely connect to the Datadog SaaS server and forward monitoring data to it. You can view and copy your Datadog API Key from the Datadog SaaS Web portal.

  1. Log into the Datadog SaaS Web portal at https://app.datadoghq.com.

    Login to Datadog
  2. Click on the Integrations link in the left frame, then click on the Agent menu item.

    Datadog Agent Integration
  3. Click on the Docker link.

    Datadog Agent Integration
  4. Select and copy the Datadog API Key. You will need to specify the Datadog API Key to the Datadog agent when deploying it.

    Copy Datadog API Key

Volume Usage

The Datadog agent container requires the following volume mounts (read-only) for gathering metrics and logs.

  • /var/run/docker.sock:/var/run/docker.sock:ro

  • /proc/:/host/proc/:ro

  • /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

Additional commands

Git client command

The Git client command was used in this Solution Guide to clone the Kubernetes State Metrics GitHub Repository. It was run on the MKE client machine as part of the setup to deploy Kubernetes State Metrics prior to deploying the Elastic stack with Kubernetes Monitoring Integration on MKE 3.2. Refer to the Git Getting Started - Installing Git web page for details on how to install the Git client if you need to install it.

curl Command

Some of the commands in this Solution Brief use the curl command.

If the curl command is not installed you can install it using the instructions below for the Linux distributions:

  • Debian/Ubuntu

    apt-get update -qq;apt-get install curl -y
    
  • CentOS/RHEL

    yum makecache fast;yum install curl -y
    
  • For other Linux distributions you can download and install it from https://curl.haxx.se/download.html.

jq command

Some of the commands in this Solution Brief use the jq command to format and display json output.

If the jq command is not installed you can install it using the instructions below for the Linux distributions:

Configure your client command shell for MKE

This Solution Brief requires you to download an MKE client bundle and configure your command shell in order to run MKE commands to your MKE Cluster. You can generate and download an MKE client bundle using the MKE Rest API or from your MKE Web UI.

Generate and download a MKE client bundle using the MKE Rest API

You can download a sample bash script named get-docker-ee-ucp-client-bundle.sh from this GitHub Repository get-docker-ee-ucp-client-bundle which uses the MKE Rest API to generate and download a MKE client bundle. You can then run the script which will download a MKE client bundle.

  1. Download the sample bash script get-docker-ee-ucp-client-bundle

  2. Export your MKE user account and password as environment variables.

    Example:

    $ export DOCKER_USER="**your-mke-user-account**"
    $ export DOCKER_PASSWORD="**your-mke-password**"
    
  3. Generate and download an MKE Client Bundle.

    Run the following command from the MKE command shell. Replace manager.example.com with the hostname or IP address of your MKE manager.

    $ ./get-docker-ee-ucp-client-bundle.sh -d manager.example.com
    

    Example:

    get-docker-ee-mke-client-bundle

  1. Configure your MKE client command shell.

    Run the following command from the MKE command shell.

    $ eval "$(<env.sh)"
    

    Example:

    Configure your MKE Client shell
  2. Test the MKE client bundle and configuration.

    Run the docker version command from the MKE client command shell.

    $ docker version --format '{{println .Server.Platform.Name}}Client: {{.Client.Version}}{{range .Server.Components}}{{println}}{{.Name}}: {{.Version}}{{end}}'
    

    Example:

    Mirantis Container Runtime Version

Generate and download a MKE client bundle from your MKE Web UI

If you prefer, you can generate and download a MKE client bundle from your MKE Web UI.

  1. Login to your MKE Cluster Web UI using your Docker ID and password.

    Login to MKE
  2. Click on your account name and then click on My Profile

    Download Client Bundle

  1. Click on the New Client Bundle button then Generate Client Bundle button.

    Download Client Bundle

  1. Locate the generated client bundle archive file and unzip it.

    Note

    The generated client bundle archive file will be downloaded to whatever folder your browser’s Download folder is configured for. You may have to move the generated client bundle archive file to the MKE client machine if you did not download it to that machine.

    Run the following command from the MKE client command shell to unzip the client bundle archive file. Substitute your generated client bundle archive file name.

    $ unzip ucp-bundle-admin.zip
    

    Example:

    Unzip Client Bundle
  2. Configure your MKE client command shell.

    Run the following command from the MKE command shell.

    $ eval "$(<env.sh)"
    

    Example:

    Configure your MKE client command shell
  3. Test the MKE client bundle and configuration.

    Run the docker version command from the MKE client command shell.

    $ docker version --format '{{println .Server.Platform.Name}}Client: {{.Client.Version}}{{range .Server.Components}}{{println}}{{.Name}}: {{.Version}}{{end}}'
    

    Example:

    Component versions

Kubernetes kubectl command

The Kubernetes kubectl command must be installed on the MKE client machine. Refer to Install and Set Up kubectl to download and install the version of the kubectl command that matches the version of Kubernetes included with the MKE version you are running. You can run the docker version command to display the version of Kubernetes installed with MKE.

Deploying Kubernetes workloads in MKE

In this Solution Brief the Datadog Agent will be deployed as a Kubernetes DaemonSet. A Kubernetes DaemonSet can be used to ensure that an application or service container runs on every node in the cluster. Monitoring and logging type applications and services typically require this type of deployment. However, in MKE the deployment of a Kubernetes DaemonSet is affected by the MKE orchestrator type that is set on the node. A node in an MKE Cluster environment contains both the Mirantis Container Runtime and the Kubernetes Agent (aka the Kubernetes kubelet). A node can be configured to run containers deployed by Docker and/or Kubernetes by setting the MKE orchestrator type on the node to one of the 3 types below.

  • Swarm - Only containers deployed using the Docker CLI and Mirantis Container Runtime API can run on this node.

  • Kubernetes - Only containers deployed using the Kubernetes CLI and Kubernetes API can run on this node.

  • Mixed - Containers deployed with Swarm or Kubernetes can run on this node. This is not recommended for production.

By default, MKE will not deploy a Kubernetes orchestrated container to a node whose orchestrator type is configured as Swarm. That presents a problem to an application or service that needs a container running on all nodes. To ensure that these types of applications and services run on all nodes, you need to deploy the application or service as a Kubernetes DaemonSet in the Kubernetes kube-system namespace. Containers deployed in the Kubernetes kube-system namespace will run on all nodes regardless of the MKE orchestrator type. The Datadog Agent needs to run on all nodes in the MKE cluster so it needs to be deployed as a Kubernetes DaemonSet in the Kubernetes kube-system namespace.

Refer to the MCR, MKE, and MSR documentation for more information on orchestration using the MCR/MKE/MSR platform.

Deploying Datadog in MKE

This example demonstrates how to deploy Datadog with Kubernetes Monitoring Integration to monitor your infrastructure on MKE.

See below for instruction on how to Install Datadog Windows Integration and the Windows Datadog Agent.

Note

The Datadog agent needs to be deployed in the kube-system namespace in the MKE environment. This will ensure that the Datadog agent container is deployed on all nodes in the MKE Cluster and that it can discover and collect Kubernetes State Metrics.

Kubernetes State Metrics

Note

If you wish to deploy Datadog with Kubernetes Monitoring Integration you need to deploy Kubernetes State Metrics prior to deploying Datadog with Kubernetes Monitoring Integration.

Kubernetes State Metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

Datadog has integration with Kubernetes State Metrics and will collect the metrics and has a dashboard which displays the Kubernetes State Metrics. In this Solution Brief Kubernetes State Metrics will be deployed and integrated with Datadog.

See also

Refer to the Kubernetes State Metrics GitHub repository for more documentation and the latest Kubernetes manifest files at this link: https://github.com/kubernetes/kube-state-metrics.

Deploy Kubernetes State Metrics

  1. The Kubernetes Manifest files used to deploy Kubernetes State Metrics are hosted and maintained on GitHub in the kube-state-metrics repository. Run the following git command below from the MKE client command shell to clone the kube-state-metrics GitHub repository.

    $ git clone https://github.com/kubernetes/kube-state-metrics.git
    
  2. Deploy Kubernetes State Metrics.

    Run the following commands from the MKE client command shell to deploy Kubernetes State Metrics.

    $ cd kube-state-metrics
    $ kubectl apply -f examples/standard
    
  3. Display the Kubernetes State Metrics Deployment.

    Run the following command from the MKE client command shell.

      $ kubectl get deployment kube-state-metrics --namespace kube-system -o wide
    
    Example:
    
    .. image:: ../_images/datadog/display_kube_state_metrics_deployment.png
       :width:
       :alt: Display Kubernetes State Metrics
    

  1. Display the pod deployed by the Kubernetes State Metrics Deployment.

    Run the following command from the MKE client command shell.

      $ kubectl get pods --namespace kube-system -o wide
    
    Example:
    
    .. image:: ../_images/datadog/display_kube_state_metrics_deployment_pod.png
       :width: 100%
       :alt: Display Kubernetes State Metrics Pod
    
  2. You can display detailed information on the Kubernetes State Metrics pod.

    Run the following kubectl describe pod command below from the MKE client command shell. Change the pod name to match the Kubernetes State Metrics pod which was deployed.

    $ kubectl describe pod --namespace kube-state-metrics-78f8b6786b-qg8wc
    
  3. Check the Kubernetes State Metrics pod logs for any errors.

    Note

    A Kubernetes State Metrics pod contains 2 containers: kube-state-metrics and addon-resizer.

    Run the following kubectl logs commands below from the MKE client command shell. Change the pod name to match the Kubernetes State Metrics pod that was deployed.

    $ kubectl logs --namespace kube-system kube-state-metrics-78f8b6786b-qg8wc --container kube-state-metrics | grep -i 'Error'
    $ kubectl logs --namespace kube-system kube-state-metrics-78f8b6786b-qg8wc --container addon-resizer | grep -i 'Error'
    
  4. Create a Kubernetes secret named dd-api-key to contain the value of your Datadog API Key.

    Run the following command from the MKE client command shell. Change the text **datadog-api-key** to your Datadog API Key.

    $ kubectl create secret generic dd-api-key --namespace kube-system --from-literal=api-key='**datadog-api-key**'
    

    Example:

    Create Kubernetes Secret

  1. Download the sample deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml included with this solution brief which contains the Kubernetes DaemonSet manifest to deploy the Datadog agent at this link: deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml

    Note

    The sample deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml included with this solution brief has been customized for MKE. Datadog provides sample Kubernetes daemonset yaml files at this link: https://github.com/DataDog/datadog-agent. For additional information on deploying the Datadog agent on Kubernetes refer to Kubernetes DaemonSet Setup.

  2. Deploy the Datadog agents with Kubernetes Monitoring Integration on all MKE Cluster nodes.

    Run the following command from the MKE client command shell.

    $ kubectl apply -f deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml
    

    Example:

    Deploy Datadog Agent DaemonSet

  1. Wait a few minutes for the Datadog environment to start up and become fully initialized.

  2. Display the Datadog Agent DaemonSet.

    Run the following command from the MKE client command shell.

    $ kubectl get daemonset --namespace kube-system datadog-agent -o wide
    

    Example:

    Display Datadog Agent DaemonSet

  1. Display the Datadog Agent DaemonSet Pods.

    Run the following command from the MKE client command shell.

    $ kubectl get pods --namespace kube-system --selector app=datadog-agent -o wide
    

    Example:

    Display Datadog Agent DaemonSet Pods

  1. You can display detailed information on a Datadog agent DaemonSet Pod.

    Run the following kubectl describe pod command below from the MKE client command shell. Change the pod name to match a Datadog agent pod that was deployed.

    $ kubectl describe pod --namespace kube-system datadog-agent-fwgft
    
  2. You can check a Datadog agent DaemonSet Pod logs for any errors.

    Run the following kubectl logs command below from the Docker Enterprise client command shell. Change the pod name to match a Datadog agent pod that was deployed.

    $ kubectl logs --namespace kube-system datadog-agent-fwgft | grep -i 'Error'
    

Install Datadog Windows Integration and the Windows Datadog Agent

Install Datadog Windows Integration

  1. Click on the Integrations link in the left frame, then click on the Integrations menu item.

    Install Datadog Windows Integration

  1. Locate and click on the Windows Service integration icon/button to install it.

  2. Click on the Configuration tab and then click on the Install Integration button. The install takes a few seconds.

  3. After the installation completes, refresh the page and scroll to the top. The Windows Service Integration will appear as Installed.

    Install Datadog Windows Integration

Install the Windows Datadog agent on a Windows MKE node

  1. Click on the Integrations link in the left frame, then click on the Agent menu item.

    Install Datadog Agent on Windows

  1. Right click the Download the Datadog Agent installer link and copy the link.

    Install Datadog Agent on Windows

  1. Run the following iwr command from a Windows Powershell command prompt on the MKE Windows Node to download the Datadog Agent installer. Replace **datadog-windows-installer-download-link** with the link you copied in the previous step.

   > iwr **datadog-windows-installer-download-link** -outfile datadog-agent-installer-windows.msi

Example:

.. image:: ../_images/datadog/install_datadog_agent_on_windows4.png
   :width: 100%
   :alt: Install Datadog Agent on Windows

  1. Run the following command from a Windows Powershell command prompt on the MKE Windows Node. Change **datadog-api-key** to your Datadog API Key.

    Note: Refer to Datadog Basic Agent Usage for Windows documentation for more details on installing the Datadog Agent on Windows including the installation parameters.

    Start-Process -Wait msiexec -ArgumentList '/qn /i datadog-agent-installer-windows.msi APM_ENABLED="true" LOGS_ENABLED="true" PROCESS_ENABLED="true" APIKEY="**datadog-api-key**"'

Example:

.. image:: ../_images/datadog/install_datadog_agent_on_windows5.png
   :width: 100%
   :alt: Install Datadog Agent on Windows

.. Note::

   The Datadog Agent is being installed silently in the
   background. It may take a few minutes for the installation to
   complete.

  1. Run the following command from a Windows Powershell command prompt on the MKE Windows Node to display the Datadog Agent Windows services.

   > get-service -name "Datadog*"

Example:

.. image:: ../_images/datadog/display_datadog_agent_on_windows.png
   :width: 100%
   :alt: Display Datadog Agent on Windows

Note

At the time of testing Windows 2019, there’s no Docker for Windows and Kubernetes for Windows support.

Display the Datadog Agents from the Datadog Monitor Web portal

You can view the deployed Datadog agent from the Datadog Web portal Host Map page.

From the Datadog Monitor Web portal click on the Infrastructure link in the left frame and then click on the Host Map link.

Display Datadog Agents

Datadog Web Interface

The Datadog Web Interface has the following modules:

Module

Description

Events

Shows a stream of the events collected by Datadog agents.

Dashboards

Allows you to view and create Dashboards that contain graphs with real-time performance metrics.

Infrastructure

Shows all of the hosts being monitoring by Datadog agents.

Monitors

Create monitors to notify you if the aggregate of a specific metric is above or below a certain threshold.

Metrics

Search, view and graph the collected metrics.

Integrations

Configure Datadog to integrate with over 200 applications, services and systems.

APM

Provides you with deep insight into your application’s performance-from automatically generated dashboards that monitor key metrics.

Notebooks

Create notebooks that combine graphs and text in a linear, cell-based format.

Logs

Search, view and graph all the logs generated by your applications, and your infrastructure.

Datadog Events

Datadog Events

Datadog Dashboards

Out of the box Datadog provides 3 system dashboards:

  1. System - Disk I/O

  2. System - Metrics

  3. System - Networking

Default Dashboards

System - Disk I/O

System - Disk I/O

System - Metrics

System - Metrics

System - Networking

System - Networking

Integration with Docker

Install the Datadog Docker Integration (Dashboard)

Datadog provides extensive Docker Integration which needs to be installed. To install the Docker Integration:

  1. Click on the Integrations link in the left frame, then click on the Integrations menu item.

  2. Locate and click on the Docker integration icon/button to install it.

    Datadog Agent Docker Integration

  1. Click on the Configuration tab and then click on the Install Integration button. The install takes a few seconds.

    Datadog Agent Docker Integration

  1. After the installation completes, refresh the page and scroll to the top. The Docker Integration will appear as Installed.

Integration with Kubernetes

Install the Kubernetes Integration (Dashboard)

Datadog provides extensive Kubernetes Integration which needs to be installed. To install the Kubernetes Integration:

  1. Follow the same process as above to install Kubernetes Integration

  2. After the installation completes, refresh the page and scroll to the top. The Kubernetes Integration will appear as Installed.

Datadog Agent Kubernetes Integration

View the Datadog Docker Dashboard

  1. Click on the Dashboards link in the left frame, followed by the Dashboard List menu item.

  2. Click on the Docker Dashboard from the dashboard list.

    View Docker Dashboard

  1. The Datadog Docker Dashboard will be displayed.

    The Datadog Docker Dashboard displays:

    • Events

      • Event timeline over the past day

      • Event stream

    • Metrics on running and stopped containers

      • Running container change over the past 5 minutes

      • Running containers

      • Stopped containers

      • Running containers by image (graph)

      • Running containers by image (list)

    • CPU Core Load

      • CPU user usage by image

      • CPU system usage by image

      • Most CPU intensive containers over the past hour

      • CPU usage by container over the past hour

    • Memory

      • RSS memory usage by image

      • SWAP usage by image, on average

      • Cache memory usage by image

      • Most RAM-intensive containers over the past hour

      • Memory by container over the past hour

    • Network

      • Average received bytes by image over the past hour

      • Average transmitted bytes by image over the past hour

      • Most network traffic intensive containers over the past hour

      • Network traffic by container over the past hour

    See also

    More details on the Datadog Docker Dashboard can be found on datadoghq.com.

    View Docker Dashboard

Infrastructure

View Docker Infrastructure

You can view the Docker Infrastructure dashboards from the Infrastructure link in the left frame:

View Docker Infrastructure

Docker Infrastructure dashboards

  1. The Host Map page provides a quick visualization of the monitored environment:

    View Docker Host Map

  1. the Infrastructure List page shows all Docker nodes being monitored (running the Datadog agent):

    View Docker Infrastructure List

    You can click on a Docker node to inspect it and get details.

    View Docker Infrastructure List Details

  1. The Containers page lists all running containers from all Docker nodes being monitored:

    View Docker Infrastructure Containers

  1. The Processes page displays a list of all running processes from all Docker nodes being monitored:

    View Docker Infrastructure Processors

Metrics

Search and graph any metrics collected by Datadog from your infrastructure environment.

Metrics Explorer

View the Datadog Kubernetes Dashboard

  1. Click on the Kubernetes Dashboard from the dashboard list.

    View Kubernetes Dashboard

  1. The Datadog Kubernetes Dashboard will be displayed.

    View Kubernetes Dashboard