Docker Enterprise Solution Guides¶

Creating a Project¶

Once a tenant is secured, create an Organization and a Project to hold code and pipeline definitions.

Configuring Source Control Management¶

Azure DevOps is capable of working with git repositories hosted in ADO itself, or in a variety of other locations such as GitHub, or on-premises servers. Create a new git repository in the ADO Project, or link to an existing repository to begin the enablement of build automation.

Automated triggers can be used between the git repository and ADO, meaning that when a git commit push occurs then an ADO Pipeline can be automatically initiate. This effectively established a continuous integration (CI) capability for source code.

Once the git repository is linked with ADO, ensure that all application code has been committed.

Triggers¶

Pipelines are initiated via Triggers, which contain the logic that determines how and when a pipeline beings execution. Triggers can be configured in a variety of ways:

• Manual will require that an operator initiate a new build
• Automatic will initiate a build any time that a commit is added to the designated source control system. To avoid builds when non-application changes are made to the repository, specify a path and/or branch to specifically target application code changes.

trigger:
  branches:
    include:
    - master
  paths:
    include:
    - app/src/*

Pools¶

When a build is triggered it is added to a queue for a given agent pool. The next available agent will then have the build assigned, and it will execute the pipeline steps.

By default ADO uses a hosted agent pool where all servers are maintained by Microsoft. Alternatively, a pool of custom agents may also be used. Please see the Agents section for more detailed information on build agent setup.

Using the Ubuntu-based hosted agent (which includes a Moby-based container runtime):

pool:
  vmImage: 'ubuntu-latest'

Using a pool of custom build agents:

pool:
  name: 'UCP Agents - Linux'

Steps¶

The steps within a pipeline define which actions are done to the source code. These actions may be defined as custom scripts or configured via pre-built tasks.

Script blocks are used to run shell code as a pipeline step. For small scripts it is fine to place these inline within the YAML. For larger scripts consider creating a scripts directory within the code repository and creating dedicated .sh, .ps1, etc. files. These files may then be called from the pipeline step without cluttering the pipeline file.

Build a Docker Image with a shell script:

scripts:
- script: |
    docker build \
      --tag $(DOCKER_REGISTRY_IMAGE):"$(git rev-parse --short HEAD)" \
      ./moby/pipeline
  displayName: 'Build Docker Image'

Build a Docker Image with a PowerShell script:

scripts:
- powershell: |
    docker build `
      --tag $(DOCKER_REGISTRY_IMAGE):"$(git rev-parse --short HEAD)" `
      .\moby\pipeline
  displayName: 'Build Docker Image'

Tasks are pre-built actions that can easily be integrated into a pipeline. A series of tasks are available out of the box from Microsoft; however, the system is also extensible through the Visual Studio Marketplace community.

Variables¶

Dockerfiles are often static assets, requiring a developer to commit code changes to adjust its behavior. Hard-coded values also impede the ability to reuse a Dockerfile across multiple contexts or image variations. The Azure DevOps platform offers variables to be defined for a given Pipeline, radically increasing the flexibility of Dockerfiles by not requiring code changes to reuse a given file.

While variables may be named any value, it is recommended to decide upon a naming convention that promotes consistency and predictability. DOCKER_ is one such convention prefix that clearly denotes that a variable is related to Docker. For example, DOCKER_REGISTRY_USERNAME would denote first that a value is related to Docker, that it is used to interact with a Registry, and that it contains an account username.

When Azure DevOps needs to use a variable that contains sensitive or secret information, a Build Secret may be employed rather than a Build Variable. When creating a variable, simply select the lock icon to convert the value to a secret. Secrets are not echoed out in logs or allowed to be seen once set. Setting the password or token used to authenticate with a Docker Registry via a DOCKER_REGISTRY_TOKEN secret would be advisable instead of a variable.

Docker Enterprise Service Account¶

Steps within an Azure DevOps Pipeline that require interaction with Docker Enterprise may use a service account model for clean separation between systems. In Mirantis Kubernetes Engine, a new user account may be created with a name such as azure-devops or similar that will serve as a service account. If using LDAP or SAML integration with a directory such as Active Directory then create an account in the external system to be synchronized into MKE.

This service account is then used whenever a pipeline needs to interact with Docker Enterprise. For example, to execute a docker push into Mirantis Secure Registry, the pipeline must first authenticate against the registry with a docker login:

- script: |
    docker login $(DOCKER_REGISTRY_FQDN) \
      --username $(DOCKER_REGISTRY_USERNAME) \
      --password $(DOCKER_REGISTRY_TOKEN)
  displayName: 'Login to Mirantis Secure Registry'

In this example the DOCKER_REGISTRY_USERNAME refers to the service account’s username, and the DOCKER_REGISTRY_TOKEN is an Access Token generated from MSR loaded into Azure DevOps as a Secret.

User accounts in Docker Enterprise utilize granular, role-based access controls (RBAC) to ensure that only the proper account has access to a given MSR repository, set of MKE nodes, etc. The service account can be directly granted permissions for pertinent MSR repositories or added to a MKE Group that inherits permissions. This system ensures that the service account has the least privileges necessary to conduct its tasks with Docker Enterprise.

A Docker Client Bundle can also be generated for this account, which can be used for continuous delivery tasks such as docker stack deploy, kubectl apply, or helm upgrade.

Build Arguments¶

The mechanism to dynamically pass a value into a Dockerfile at docker build time is the --build-arg flag. A variable or secret can be used with the flag to change a build outcome without committing a code change into the source control system. To utilize the flag, we add an ARG line to our Dockerfile for each variable to be passed.

For example, to dynamically expose a port with in the Dockerfile we would adjust:

FROM mcr.microsoft.com/dotnet/core/sdk:2.1
EXPOSE 80
WORKDIR /app

to include an ARG

FROM mcr.microsoft.com/dotnet/core/sdk:2.1
ARG DOCKER_IMAGE_PORT=80
EXPOSE ${DOCKER_IMAGE_PORT}
WORKDIR /app

Note that we set a default value by including ``=80``; this value
will be used if a dynamic value is not passed in at build time.

A base image may also be made into a dynamic value, however the ARG must be placed outside of the FROM statement. For example, to adjust the following Dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:2.1
EXPOSE 80
WORKDIR /app

Place the ARG at the beginning of the file and use the variable name in the FROM statement:

ARG BASE_IMAGE='mcr.microsoft.com/dotnet/core/sdk:2.1'

FROM ${BASE_IMAGE}
EXPOSE 80
WORKDIR /app

Positioning the ARG outside of the FROM statement(s) places it in a higher scope than within any specific stage.

Using the ARG and --build-arg pattern is useful to easily patch a given image when improvements are made to its base image. Adjusting a build variable and initiating a build brings the newer base image tag in without requiring a formal code commit.

Labels¶

Metadata may be added to a Dockerfile via the LABEL keyword. Labels can help designate particular application owners, points of contact, or extraneous characteristics that would benefit from embedding within an image.

Some common settings in Dockerfiles include:

FROM mcr.microsoft.com/dotnet/core/sdk:2.1
LABEL IMAGE_OWNER='Moby Whale <mobyw@docker.com>'
LABEL IMAGE_DEPARTMENT='Digital Marketing'
LABEL IMAGE_BUILT_BY='Azure DevOps'
EXPOSE 80
WORKDIR /app

Combine LABEL with ARG to dynamically pass metadata values into an image at build time.

FROM mcr.microsoft.com/dotnet/core/sdk:2.1
ARG COMMIT_ID=''
LABEL GIT_COMMIT_ID=${COMMIT_ID}

docker build `
  --build-arg COMMIT_ID="$(git rev-parse --short HEAD)" `
  --tag $(DOCKER_REGISTRY_IMAGE):"$(git rev-parse --short HEAD)" `
  .

Values for LABEL can be viewed in a container registry such as Mirantis Secure Registry (MSR), or from the Docker CLI:

$ docker inspect moby:1
[
    {
        ...
        "Config": {
            ...
            "Labels": {
                "GIT_COMMIT_ID": "421b895"
            }
        }
        ...
    }
]

Multi-Stage Builds¶

Dockerfiles originally functioned as a single “stage”, where all steps took place in the same context. All libraries and frameworks necessary for the Dockerfile to build had to be loaded in, bloating the size of the resulting images. Much of this image size was used during the build phase, but was not necessary for the application to properly run; for example, after an application compiles it does not necessarily have to have the compiler or SDK within the image to run.

The introduction of “multi-stage builds” introduced splitting out of individual “stages” within one physical Dockerfile. In a build system such as Azure DevOps, we can define a “builder” stage with a base layer containing all necessary compilation components, and then a second, lightweight “runtime” stage devoid of hefty SDKs and compilation tooling. This last stage becomes the built image, with the builder stage serving only as a temporary intermediary.

#=======================================================
# Stage 1: Use the larger SDK image to compile .NET code
#=======================================================
FROM mcr.microsoft.com/dotnet/core/sdk:2.1 AS build
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.sln .
COPY aspnetapp/*.csproj ./aspnetapp/
RUN dotnet restore

# copy everything else and build app
COPY aspnetapp/. ./aspnetapp/
WORKDIR /app/aspnetapp
RUN dotnet publish -c Release -o out

#=========================================================
# Stage 2: Copy built artifact into the slim runtime image
#=========================================================
FROM mcr.microsoft.com/dotnet/core/aspnet:2.1 AS runtime
EXPOSE 80
WORKDIR /app
COPY --from=build /app/aspnetapp/out ./
ENTRYPOINT ["dotnet", "aspnetapp.dll"]

Using multi-stage builds in your pipelines has considerable impact to the speed and efficiency of builds, and the resulting file size of the built container image. For a .NET Core application, the aspnet base layer without the sdk components is substantially smaller:

$ docker image ls \
  --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}' \
  --filter=reference='mcr.microsoft.com/dotnet/core/*:*'

REPOSITORY                              TAG                 SIZE
mcr.microsoft.com/dotnet/core/sdk       2.1                 1.74GB
mcr.microsoft.com/dotnet/core/sdk       2.2                 1.74GB
mcr.microsoft.com/dotnet/core/aspnet    2.1                 253MB
mcr.microsoft.com/dotnet/core/runtime   2.1                 180MB

For more information please see the Docker Documentation for
`Multi-Stage Builds <TODO>`__.

Hosted Agents¶

The easiest way to execute builds on Azure DevOps is by using the hosted build agents. These VM-based agent pools come in a variety of operating systems and have Docker available. Pipeline steps such as docker build are available without additional configuration.

Being a hosted environment, there is minimal ability to customize the SDKs, tooling, and other components within the virtual machine. A pipeline step could be used to install needed software via approaches such as apt-get or chocolatey, however doing so may add substantial time to each build considering the VMs are wiped after the pipeline completes.

The use of multi-stage builds decreases this dependency on the hosted build environment, as all necessary components for a container image should be embedded into the “builder” Dockerfile stage. To adjust the container runtime itself, for example to use a supported Docker Enterprise engine, a self-hosted agent is necessary.

Self-Hosted Agents¶

Running a self-hosted agent provides the ability to customize every facet of the build environment. Container runtimes can be customized, specific SDKs and compilers added, and for scenarios where project requirements restrict cloud-based agents a self-hosted agent can enable on-premises builds.

The downside is that infrastructure must be deployed and maintained to host the agents. These severs or virtual machines install a software agent, which connects to a specific Azure DevOps tenant.

The software agent can also run within a Docker container. Deploying containerized agents provides an array of benefits compared to traditional VM-based build agents including:

• Utilize existing container clusters, such as Docker Enterprise, rather than dedicated infrastructure
• Easily scale up and down the number of build agent instances with the Kubernetes or Swarm orchestrators
• Minimal assets to maintain when multi-stage builds are used as the containerized agent only needs access to the host’s Docker daemon

If building traditional software that does not run in a container, then simply install the Azure DevOps agent and connect to a tenant. For containerized applications that require a docker build pipeline step, the Docker CLI is installed within the container so that it may connect to the host’s Docker Daemon. This connection executes the build on the host but controlled from a containerized build agent.

To make this connection on Linux, a Docker Volume is used to mount the host’s daemon:

docker run --volume /var/run/docker.sock:/var/run/docker.sock ...

And on Windows a Named Pipe is used via a similar approach:

docker run --volume \\.\pipe\docker_engine:\\.\pipe\docker_engine ...

Note that the ability to use Named Pipes `was
introduced <https://blog.docker.com/2017/09/docker-windows-server-1709/>`__
in Windows Server 1709 (SAC), and Windows Server 2019 (LTSC)

These volume mounts allow the Docker CLI to execute commands such as docker build within the container, and have the action executed on the host.

#=========================================================
# Stage 1: Download Docker CLI binary
#=========================================================
FROM alpine:latest AS dockercli

ARG DOCKER_BRANCH=test
ARG DOCKER_VERSION=19.03.0-beta4

RUN wget --output docker.tgz https://download.docker.com/linux/static/${DOCKER_BRANCH}/x86_64/docker-${DOCKER_VERSION}.tgz && \
    tar -zxvf docker.tgz && \
    chmod +x docker/docker

#=========================================================
# Stage 2: Download kubectl binary
#=========================================================
FROM alpine:latest AS kubectl

ARG KUBECTL_VERSION=v1.14.1

RUN wget --output ./kubectl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl && \
    chmod +x ./kubectl

#=========================================================
# Stage 3: Download Helm binary
#=========================================================
FROM alpine:latest AS helm

ARG HELM_VERSION=v2.13.1

RUN wget --output helm.tar.gz https://storage.googleapis.com/kubernetes-helm/helm-${HELM_VERSION}-linux-amd64.tar.gz && \
    tar -zxvf helm.tar.gz && \
    chmod +x linux-amd64/helm

#=========================================================
# Stage 4: Setup Azure Pipelines remote agent
# Documented at https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#linux
#=========================================================
FROM ubuntu:16.04

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
  ca-certificates \
  curl \
  jq \
  git \
  iputils-ping \
  libcurl3 \
  libicu55 \
  zip \
  unzip \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /azp

COPY ./start.sh .

# User is required for the Docker Socket
USER root

# Copy binaries from earlier build stages
COPY --from=dockercli docker/docker /usr/local/bin/docker
COPY --from=kubectl ./kubectl /usr/local/bin/kubectl
COPY --from=helm /linux-amd64/helm /usr/local/bin/helm

CMD ["./start.sh"]

# escape=`

#=========================================================
# Stage 1: Download Docker CLI binary
#=========================================================
FROM mcr.microsoft.com/windows/servercore:ltsc2019 AS dockercli

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

RUN Invoke-WebRequest `
      -OutFile docker.zip `
      -Uri https://download.docker.com/components/engine/windows-server/18.09/docker-18.09.6.zip `
      -UseBasicParsing; `
    Expand-Archive `
      -DestinationPath 'C:\' `
      -Force `
      -Path docker.zip;
#=========================================================
# Stage 2: Download Azure DevOps Pipelines Agent
#=========================================================
FROM mcr.microsoft.com/windows/servercore:ltsc2019 AS adoagent
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
ENV ADO_AGENT_URL='https://vstsagentpackage.azureedge.net/agent/2.148.2/vsts-agent-win-x64-2.148.2.zip'
RUN Invoke-WebRequest `
      -OutFile C:\agent.zip `
      -Uri $env:ADO_AGENT_URL `
      -UseBasicParsing; `
    Expand-Archive `
      -Destination C:\agent `
      -Force `
      -Path agent.zip;
#=========================================================
# Stage 3: Download ServiceMonitor
# https://github.com/microsoft/IIS.ServiceMonitor
#=========================================================
FROM mcr.microsoft.com/windows/servercore:ltsc2019 AS servicemonitor
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
ENV SERVICE_MONITOR_VERSION='2.0.1.3'
RUN Invoke-WebRequest `
      -OutFile C:\ServiceMonitor.exe `
      -Uri "https://dotnetbinaries.blob.core.windows.net/servicemonitor/$Env:SERVICE_MONITOR_VERSION/ServiceMonitor.exe" `
      -UseBasicParsing;
#=========================================================
# Stage 4: Setup Azure Pipelines remote agent
#=========================================================
FROM mcr.microsoft.com/windows/servercore:ltsc2019 AS runtime
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
WORKDIR C:\agent
# Setup general tools via the Scoop package manager
RUN Invoke-Expression (New-Object Net.WebClient).DownloadString('https://get.scoop.sh'); `
    scoop install git;
# Setup Azure Pipelines Agent
COPY --from=adoagent C:\agent C:\agent
# Setup Docker CLI
COPY --from=dockercli C:\docker C:\docker
# Setup ServiceMonitor
COPY --from=servicemonitor C:\ServiceMonitor.exe C:\ServiceMonitor.exe
# Update path variable
RUN $env:PATH = 'C:\docker;' + $env:PATH; `
    Set-ItemProperty `
      -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\' `
      -Name Path `
      -Value $env:PATH;

# Copy startup script into container
COPY start.ps1 .

# Run startup script on initialization
CMD .\start.ps1

# https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows

# ===============================
# Check if required variables are present
# ===============================
if (!$env:AZP_URL) { Write-Host "The AZP_URL environment variable is null. Please adjust before continuing"; exit 1; }
if (!$env:AZP_TOKEN) { Write-Host "The AZP_TOKEN environment variable is null. Please adjust before continuing"; exit 1; }
if (!$env:AZP_POOL) { $env:AZP_POOL='Default' }

# ===============================
# Configure Azure Pipelines Agent
# ===============================
if(!(Test-Path -Path C:\agent\_work )) {
  Write-Output "No previous agent configuration detected. Configuring agent."
  .\config.cmd `
    --acceptTeeEula `
    --auth PAT `
    --pool "${env:AZP_POOL}" `
    --replace `
    --runAsService `
    --token "${env:AZP_TOKEN}" `
    --unattended `
    --url "${env:AZP_URL}" `
    --windowsLogonAccount "NT AUTHORITY\SYSTEM"

}

# ==============================
# Run Azure Pipelines Agent with ServiceMonitor
# ==============================
C:\ServiceMonitor.exe (Get-Service vstsagent*).Name

Deploying with Helm¶

In the Kubernetes world, Helm is a popular tool for deploying and managing the life cycle of container workloads. A Helm “Chart” is created via the helm create command, and then values are adjusted to match a given application’s needs. Docker Enterprise is a CNCF Certified distribution of Kubernetes and works seamlessly with Helm.

Azure DevOps Pipelines can interface with Docker Enterprise via Helm by having the kubectl binary installed in the build agent. This command line tool is then further configured to work with MKE through a Docker Client Bundle, which establishes a secure connection context between kubectl and MKE. Once established, standard Helm commands may be issued to update a running Helm workload.

In the following example, a formal deploy stage is created in the Azure DevOps Pipeline that depends on the successful completion of a build stage. A Client Bundle is then downloaded from the Azure DevOps Secure File Library, unzipped, and sourced. The helm upgrade command then updates the declared image tag to the recently build tag from MSR with --set "image.tag=$(git rev-parse --short HEAD)". Helm then gracefully handles the upgrade process for the running image.

- stage: deploy
  displayName: Deploy to Cluster
  dependsOn:
  - build
  jobs:
  - job: helm
    displayName: 'Deploy Container with Helm'
    pool:
      name: 'Shared SE - Linux'
      demands:
      - agent.os -equals Linux
      - docker
    steps:
    - task: DownloadSecureFile@1
      inputs:
        secureFile: 'ucp-bundle-azure-devops.zip'

    - script: |
        # Unzip Docker Client Bundle from UCP
        unzip \
          -d $(Agent.TempDirectory)/bundle \
          $(Agent.TempDirectory)/ucp-bundle-azure-devops.zip
      displayName: 'Setup Docker Client Bundle'

    - script: |
        # Connect Helm to cluster via Docker Client Bundle
        cd $(Agent.TempDirectory)/bundle
        eval "$(<env.sh)"

        # Update deployment with Helm
        cd $(Build.SourcesDirectory)
        helm upgrade \
          web \
          ./pipelines/helm/techorama \
          --set "image.tag=$(git rev-parse --short HEAD)" \
          --tiller-namespace se-stevenfollis
      displayName: 'Update application with Helm'

Git client command¶

The Git client command was used in this Solution Brief to clone the Kubernetes State Metrics GitHub Repository. It was run on the Docker Enterprise client machine as part of the setup to deploy Kubernetes State Metrics prior to deploying the Elastic stack with Kubernetes Monitoring Integration on Docker Enterprise 3.0. Refer to the Git Getting Started - Installing Git web page for details on how to install the Git client if you need to install it.

curl Command¶

Some of the commands in this Solution Brief use the curl command.

If the curl command is not installed you can install it using the instructions below for the Linux distributions:

• Debian/Ubuntu

  apt-get update -qq;apt-get install curl -y
  

• CentOS/RHEL

  yum makecache fast;yum install curl -y
  

• For other Linux distributions you can download and install it from https://curl.haxx.se/download.html.

jq command¶

Some of the commands in this Solution Brief use the jq command to format and display json output.

If the jq command is not installed you can install it using the instructions below for the Linux distributions:

• Debian/Ubuntu

  $ apt-get update -qq;apt-get install jq -y
  

• CentOS/RH

  $ yum makecache fast;yum install jq -y
  

• For other Linux distributions you can download and install it from https://github.com/stedolan/jq/wiki/Installation.

Generate and download a MKE client bundle using the MKE Rest API¶

You can download a sample bash script named get-docker-ee-ucp-client-bundle.sh from this GitHub Repository get-docker-ee-ucp-client-bundle which uses the Docker Enterprise MKE Rest API to generate and download a Docker Enterprise MKE client bundle. You can then run the script which will download a Docker Enterprise MKE client bundle.

1. Download the sample bash script get-docker-ee-ucp-client-bundle

2. Export your Docker Enterprise MKE user account and password as environment variables.

   Example:

   $ export DOCKER_USER="**your-docker-ee-ucp-user-account**"
   $ export DOCKER_PASSWORD="**your-docker-ee-ucp-password**"
   

3. Generate and download a Docker Enterprise MKE Client Bundle.

   Run the following command from the Docker Enterprise command shell. Replace manager.example.com with the hostname or IP address of your Docker Enterprise MKE manager.

   $ ./get-docker-ee-ucp-client-bundle.sh -d manager.example.com
   

   Example:

   

4. Configure your Docker Enterprise client command shell.

   Run the following command from the Docker Enterprise command shell.

   $ eval "$(<env.sh)"
   

   Example:

   

5. Test the Docker Enterprise MKE client bundle and configuration.

   Run the docker version command from the Docker Enterprise client command shell.

   $ docker version --format '{{println .Server.Platform.Name}}Client: {{.Client.Version}}{{range .Server.Components}}{{println}}{{.Name}}: {{.Version}}{{end}}'
   

   Example:

   

Generate and download a MKE client bundle from your MKE Web UI¶

If you prefer, you can generate and download a Docker Enterprise MKE client bundle from your Docker Enterprise MKE Web UI.

1. Login to your Docker Enterprise MKE Cluster Web UI using your Docker ID and password.

   

2. Click on your account name and then click on My Profile

   

3. Click on the New Client Bundle button then Generate Client Bundle button.

   

4. Locate the generated client bundle archive file and unzip it.

   Note

   The generated client bundle archive file will be downloaded to whatever folder your browser’s Download folder is configured for. You may have to move the generated client bundle archive file to the Docker Enterprise client machine if it is a different machine than the machine you downloaded it on.

   Run the following command from the Docker Enterprise client command shell to unzip the client bundle archive file. Substitute your generated client bundle archive file name.

   $ unzip ucp-bundle-admin.zip
   

   Example:

   

5. Configure your Docker Enterprise client command shell.

   Run the following command from the Docker Enterprise command shell.

   $ eval "$(<env.sh)"
   

   Example:

   

6. Test the Docker Enterprise MKE client bundle and configuration.

   Run the docker version command from the Docker Enterprise client command shell.

   $ docker version --format '{{println .Server.Platform.Name}}Client: {{.Client.Version}}{{range .Server.Components}}{{println}}{{.Name}}: {{.Version}}{{end}}'
   

   Example:

   

Deploy Kubernetes State Metrics¶

1. The Kubernetes Manifest files to deploy Kubernetes State Metrics are hosted and maintained on GitHub in the kube-state-metrics Repository. Run the following git command below from the Docker Enterprise client command shell to clone the kube-state-metrics GitHub repository.

   $ git clone https://github.com/kubernetes/kube-state-metrics.git
   

2. Deploy Kubernetes State Metrics.

   Run the following commands from the MKE client command shell to deploy Kubernetes State Metrics.

   $ cd kube-state-metrics
   $ kubectl apply -f examples/standard
   

3. Display the Kubernetes State Metrics Deployment.

   Run the following command from the MKE client command shell.

     $ kubectl get deployment kube-state-metrics --namespace kube-system -o wide
   
   Example:
   
   .. image:: ../_images/datadog/display_kube_state_metrics_deployment.png
      :width:
      :alt: Display Kubernetes State Metrics
   

4. Display the pod deployed by the Kubernetes State Metrics Deployment.

   Run the following command from the MKE client command shell.

     $ kubectl get pods --namespace kube-system -o wide
   
   Example:
   
   .. image:: ../_images/datadog/display_kube_state_metrics_deployment_pod.png
      :width: 100%
      :alt: Display Kubernetes State Metrics Pod
   

5. You can display detailed information on the Kubernetes State Metrics pod.

   Run the following kubectl describe pod command below from the MKE client command shell. Change the pod name to match the Kubernetes State Metrics pod which was deployed.

   $ kubectl describe pod --namespace kube-state-metrics-78f8b6786b-qg8wc
   

6. Check the Kubernetes State Metrics pod logs for any errors.

   Note

   A Kubernetes State Metrics pod contains 2 containers: kube-state-metrics and addon-resizer.

   Run the following kubectl logs commands below from the Docker MKE client command shell. Change the pod name to match the Kubernetes State Metrics pod which was deployed.

   $ kubectl logs --namespace kube-system kube-state-metrics-78f8b6786b-qg8wc --container kube-state-metrics | grep -i 'Error'
   $ kubectl logs --namespace kube-system kube-state-metrics-78f8b6786b-qg8wc --container addon-resizer | grep -i 'Error'
   

7. Create a Kubernetes secret named dd-api-key to contain the value of your Datadog API Key.

   Run the following command from the Docker Enterprise client command shell. Change the text **datadog-api-key** to your Datadog API Key.

   $ kubectl create secret generic dd-api-key --namespace kube-system --from-literal=api-key='**datadog-api-key**'
   

   Example:

   

8. Download the sample deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml included with this solution brief which contains the Kubernetes DaemonSet manifest to deploy the Datadog agent at this link: deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml

   Note

   The sample deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml included with this solution brief has been customized for Docker Enterprise. Datadog provides sample Kubernetes daemonset yaml files at this link: https://github.com/DataDog/datadog-agent. For additional information on deploying the Datadog agent on Kubernetes refer to: Kubernetes DaemonSet Setup.

9. Deploy the Datadog agents with Kubernetes Monitoring Integration on all Docker Enterprise MKE Cluster nodes.

   Run the following command from the Docker Enterprise client command shell.

   $ kubectl apply -f deploy-datadog-agent-with-kubernetes-monitoring-integration.yaml
   

   Example:

   

10. Wait a few minutes for the Datadog environment to start up and become fully initialized.

11. Display the Datadog Agent DaemonSet.

    Run the following command from the Docker Enterprise client command shell.

    $ kubectl get daemonset --namespace kube-system datadog-agent -o wide
    

    Example:

    

12. Display the Datadog Agent DaemonSet Pods.

    Run the following command from the Docker Enterprise client command shell.

    $ kubectl get pods --namespace kube-system --selector app=datadog-agent -o wide
    

    Example:

    

13. You can display detailed information on a Datadog agent DaemonSet Pod.

    Run the following kubectl describe pod command below from the Docker Enterprise client command shell. Change the pod name to match a Datadog agent pod which was deployed.

    $ kubectl describe pod --namespace kube-system datadog-agent-fwgft
    

14. You can check a Datadog agent DaemonSet Pod logs for any errors.

    Run the following kubectl logs command below from the Docker Enterprise client command shell. Change the pod name to match a Datadog agent pod which was deployed.

    $ kubectl logs --namespace kube-system datadog-agent-fwgft | grep -i 'Error'
    

Install Datadog Windows Integration¶

1. Click on the Integrations link in the left frame, then click on the Integrations menu item.

   

2. Locate and click on the Windows Service integration icon/button to install it.

3. Click on the Configuration tab and then click on the Install Integration button. The install takes a few seconds.

4. After the installation completes, refresh the page and scroll to the top. The Windows Service Integration will appear as Installed.

   

Install the Windows Datadog agent on a Windows Docker Enterprise MKE node¶

1. Click on the Integrations link in the left frame, then click on the Agent menu item.

   

2. Right click the Download the Datadog Agent installer link and copy the link.

   

3. Run the following iwr command from a from a Windows Powershell command prompt on the Docker Enterprise Windows Node to download the Datadog Agent installer. Replace **datadog-windows-installer-download-link** with the link you copied in the previous step.

   > iwr **datadog-windows-installer-download-link** -outfile datadog-agent-installer-windows.msi

Example:

.. image:: ../_images/datadog/install_datadog_agent_on_windows4.png
   :width: 100%
   :alt: Install Datadog Agent on Windows

4. Run the following command from a Windows Powershell command prompt on the Docker Enterprise Windows Node. Change **datadog-api-key** to your Datadog API Key.

   Note: Refer to Datadog Basic Agent Usage for Windows documentation for more details on installing the Datadog Agent on Windows including the installation parameters.

    Start-Process -Wait msiexec -ArgumentList '/qn /i datadog-agent-installer-windows.msi APM_ENABLED="true" LOGS_ENABLED="true" PROCESS_ENABLED="true" APIKEY="**datadog-api-key**"'

Example:

.. image:: ../_images/datadog/install_datadog_agent_on_windows5.png
   :width: 100%
   :alt: Install Datadog Agent on Windows

.. Note::

   The Datadog Agent is being installed silently in the
   background. It may take a few minutes for the installation to
   complete.

5. Run the following command from a Windows Powershell command prompt on the Docker Enterprise Windows Node to display the Datadog Agent Windows services.

   > get-service -name "Datadog*"

Example:

.. image:: ../_images/datadog/display_datadog_agent_on_windows.png
   :width: 100%
   :alt: Display Datadog Agent on Windows

Datadog Events¶

Datadog Dashboards¶

Out of the box Datadog provides 3 system dashboards:

1. System - Disk I/O
2. System - Metrics
3. System - Networking

System - Disk I/O¶

System - Metrics¶

System - Networking¶

Integration with Docker¶

Integration with Kubernetes¶

Install the Kubernetes Integration (Dashboard)¶

Datadog provides extensive Kubernetes Integration which needs to be installed. To install the Kubernetes Integration:

1. Follow the same process as above to install Kubernetes Integration
2. After the installation completes, refresh the page and scroll to the top. The Kubernetes Integration will appear as Installed.

Infrastructure¶

View Docker Infrastructure¶

You can view the Docker Infrastructure dashboards from the Infrastructure link in the left frame:

Metrics¶

Search and graph any metrics collected by Datadog from your infrastructure environment.

Product Benefits¶

• Auto-discovery - Dynatrace Smartscape® technology automatically detects and displays all the components and dependencies that comprise your environment, providing you with a real-time blueprint of your application architecture.
• Auto-baselining - Dynatrace uses AI to understand the difference between desirable and unwanted behaviors emerging from each component. It can also dive deep into the code to uncover methods being invoked and how they contribute to the overall performance of the system.
• Auto-problem analysis - Dynatrace rapidly identifies failures, component involvement, and root causes. Machine learning helps evaluate issues, determine if it warrants an alert, and provide valuable insights for better business decisions.

Dynatrace’s main features¶

• Hybrid multi-cloud, container, and infrastructure monitoring
• Support for native apps, microservices and containers
• Full stack discovery
• Single transaction analysis
• User behavior analytics
• Automated root-cause analysis
• Docker Swarm and Kubernetes integration
• SaaS or on-premise deployment
• Deploy, configure, and manage with zero touch
• Intuitive dashboards pre-configured with self-learning
• Open ecosystem with seamless integration along with APIs and platform extensions

Dynatrace Supported Technologies¶

Dynatrace supports are large array of technologies. Some of the most popular ones are:

AMP, ASPNET Core, AWS CodePipeline, AWS Elastic Load Balancing (ELB), AWS Lambda, ActiveMQ, Akka, Amazon DynamoDB, Amazon EC2, Amazon Elastic Block Store (EBS), Amazon RDS, Amazon S3, Amazon Web Services, Android, Android app monitoring, AngularJS, Ansible, Apache Cordova, Apache HTTP Server, Azure App Service, Azure Container Services (AKS), Azure Virtual Machines, Azure Web Apps, Bitbucket, C, C++, CakePHP, Cassandra, Citrix, Cloud Foundry, CouchDB, Couchbase, DC/OS, Demandware, Docker, Drupal, Elasticsearch, Erlang, Eureka, GlassFish, Golang, Google Cloud Platform, GraalVM, Grails, Hadoop, Heroku, HipChat, Hyper-V, Hystrix, IBM Bluemix, IBM Cloud, IBM Commerce, IBM HTTP Server, IBM WebSphere, IBM z/OS, JBoss, Java, Jetty, Jira, Joomla, Jython, Kubernetes, Laravel, Magento, Memcached, Mesos, Microsoft Azure, Microsoft IIS, Microsoft SharePoint, MongoDB, MySQL, .NET, .NET Core, Netflix OSS, Netty, Nginx, NodeJS, OpenStack, OpenTracing, Openshift, OpsGenie, Oracle Cloud, Oracle DB, Oracle HTTP Server, Oracle WebLogic, Oracle eBusiness, PHP, PagerDuty, Perl, Pivotal, Pivotal Cloud Foundry, Play Framework, PostgreSQL, Python, RabbitMQ, Reactjs, Red Hat, Redis, Ruby, SAP, SAP Cloud Platform, SAP Hybris, SQL Server, Salesforce Commerce Cloud, Scala, ServiceNow, Sitecore, Slack, Spark, Splunk, Spring, Symfony, TomEE, Tomcat, Trello, VMware, VMware Cloud on AWS, Varnish, VictorOps, Webhooks, Wildfly, Windows Messaging, WordPress, iOS, iOS, jQuery, xMatters

Dynatrace Monitoring environment¶

The Dynatrace Monitoring environment consists of a Dynatrace server, Dynatrace agents and Dynatrace Activate (proxy).

Install the Linux Dynatrace agent on a Linux Docker Enterprise MKE node¶

1. From the Dynatrace Web UI, navigate to the Deploy Dynatrace page by clicking on the Deploy Dynatrace link (bottom left). Then click on the Start installation button.

   

2. From the Download Dynatrace OneAgent page click on the Linux button.

   

3. Copy the wget command which downloads the Dynatrace agent installation bits by clicking on the Copy button to the right of the command.

   

4. Paste the wget command into a Linux command prompt and run it with root authority (sudo).

   

5. Copy the command to install the Dynatrace agent by clicking on the Copy button to the right of the command.

   

6. Paste the command into a Linux command prompt and run it with root authority (sudo).

   

If Docker Enterprise was installed prior to installing the Dynatrace agent you will need to reboot the machine.

Repeat the above Dynatrace agent installation on all of the Linux Docker nodes.

Install the Windows Dynatrace agent on a Windows Docker Enterprise MKE node¶

1. From the Dynatrace Web UI, navigate to the Deploy Dynatrace page by clicking on the Deploy Dynatrace link (bottom left). Then click on the Start installation button.

   

2. From the Download Dynatrace OneAgent page click on the Windows button.

   

3. Right click on the Download agent.exe button and copy the Link to download the installation bits.

   

4. Run the following iwr command from a Powershell command prompt to download the Windows Dynatrace agent installation bits. Substitute the text dynatrace-oneagent-windows-download-link with the Windows Dynatrace agent download Link you copied in the previous step.

   > iwr -outfile install_oneagent.exe -uri '**dynatrace-oneagent-windows-download-link**'

Example:

.. image:: ../_images/dynatrace/run_dynatrace_oneagent_installer_iwr_command.png
   :width: 100%
   :alt: Run the iwr command

5. Run the following install_oneagent.exe command from a Powershell command prompt to install the Windows Dynatrace agent.

    .\install_oneagent.exe /quiet /norestart

Example:

.. image:: ../_images/dynatrace/run_dynatrace_oneagent_installer_windows_command.png
   :width: 100%
   :alt: Run the iwr command

.. Note::

   The **install\_oneagent.exe** command runs detached in the background.

6. Wait about a minute and then run the following command from a Powershell command prompt to display the Dynatrace agent windows service.

   > net start | findstr /c:"Dynatrace OneAgent"

Example:

.. image:: ../_images/dynatrace/display_dynatrace_oneagent_windows_service.png
   :width: 100%
   :alt: Run the iwr command

If Docker Enterprise was installed prior to installing the Dynatrace agent you will need to reboot the machine.

Repeat the above Dynatrace agent installation on all of the Windows Docker nodes.

Display the Dynatrace agents from the Dynatrace Web UI¶

You can display the Dynatrace agents from the Dynatrace Web UI Deployment Status page.

1. From the Dynatrace Web UI, click on the Deployment Status link.

   

2. You can click on a Dynatrace agent to get more details.

   

3. More details are displayed which includes the processes and containers being monitored by the selected Dynatrace agent.

   

Dashboards & reports¶

• Dashboards - allow you to create your own custom dashboards to monitor your infrastructure and applications. Dynatrace provides a default home dashboard that provides a quick overview of your system’s health.
• Create custom chart - Custom charts enable you to analyze any combination of monitoring metrics directly on your dashboards.
• Reports - Dynatrace provides Service quality and Availability reports. Service quality reports are generated each week on Sundays at midnight so that when you start your work week each Monday morning you’ll find a new report ready for your review. Availability reports are based on web checks that you have running in Dynatrace.

Analyze¶

• Problems - Dynatrace automatically detects performance anomalies in your applications, services, and infrastructure. Dynatrace “problems” are used to report and alert on abnormal situations, such as performance degradations, improper functionality, or lack of availability (i.e., problems represent anomalies in baseline system performance).
• User sessions - A user session, sometimes known as a “visit,” is a group of user actions that are performed in your web application during a limited period of time.
• Log files - Dynatrace automatically discovers all log files on your monitored hosts and processes. You can then browse, search and filter for information in the log files.
• Smartscape topology - Visualize your application topology in an interactive infographic.
• Diagnostic tools - Dynatrace provide diagnostic tools for detecting and analyzing performance problems in your application environment.

Monitor¶

• Applications - Dynatrace monitors and visualizes application performance, provides all performance metrics in real time and detects and diagnoses problems automatically.
• Synthetic - Dynatrace allows you to define Synthetic monitors (simulated user sessions) that alert you when an application is inaccessible or when performance degrades below baseline performance.
• Transactions & services - Dynatrace captures every transaction from all applications and services.
• Databases - Dynatrace detects and monitors all databases, captures every database statement and displays detailed health metrics for each statement.
• Hosts - Dynatrace automatically monitors the state of a host machine along with the running processes and containers.
• Network - Dynatrace monitors and visualizes process-specific network performance metrics like requests and retransmissions to proactively identify connection issues.
• Technologies - Dynatrace detects and monitors over 120 technologies (clouds, applications, databases, web, integrations, etc.)
• VMware - Dynatrace monitors and visualizes your VMware environments to understand the relationships between components such as ESXi hosts and your vCenter platform as a whole.
• AWS - Dynatrace monitors and visualizes services running in the Amazon Web Services Cloud.
• Azure - Dynatrace monitors and visualizes services running in the Microsoft Azure Cloud.
• Docker - Dynatrace offers full-featured Docker monitoring and provides you with a complete picture of your Dockerized environments.

Manage¶

• Deploy Dynatrace - Simple wizard to walk you through deploying the Dynatrace agent.
• Deployment status - Display status of the deployed Dynatrace agents.
• Settings - Configure Dynatrace monitoring settings from the Dynatrace Web UI.

Deep monitoring of a Tomcat application¶

Deep monitoring of a MySQL Database¶

Deep monitoring of a network¶

Monitor AWS¶

You can configure Dynatrace to perform monitoring of services running in the AWS Cloud.

• Elastic Compute Cloud (EC2)
• Elastic Block Store (EBS)
• Elastic Load Balancer (ELB)
• Relational Database Service (RDS)
• DynamoDB
• Lambda

Monitor AWS Services¶

Monitor Docker nodes¶

You can display an overview of all Docker nodes and you can drill down to a specific Docker node to monitor CPU, Memory, Network and Disk utilization.

You can drill down to a specific Docker container.

On ESX¶

Install Docker Enterprise 3.0 (MKE and MSR) and configure worker node orchestration to Kubernetes.

Highlevel Cluster diagram

Login to Docker Enterprise MKE Cluster Web UI using Docker username and password.

Docker Enterprise MKE Dashboard

Configure Node Orchestration from Swarm to Kubernetes. Double click on highlighted node.

Click on the Edit Node on upper right hand, and change Orchestrator Type to kubernetes

New display of nodes will show Orchestrator Type as kubernetes for workers nodes

Generate and download a Docker Enterprise MKE client bundle from MKE Web UI¶

1. Click on your account name and then click on My Profile

   

2. Click on the New Client Bundle button then Generate Client Bundle button.

   

3. Locate the generated client bundle archive file and unzip it.

   Note

   The generated client bundle archive file will be downloaded to whatever folder your browser’s Download folder is configured for. You may have to move the generated client bundle archive file to the Docker client machine if it is a different machine than the machine it was downloaded to.

   Run the following command from the Docker Enterprise client command shell to unzip the client bundle archive file.

   $ unzip ucp-bundle-admin.zip
   

   Example:

   

4. Configure Docker Enterprise client command shell.

   Run the following command from the Docker Enterprise command shell.

   $ eval "$(<env.sh)"
   

   Example:

   

5. Test the Docker Enterprise MKE client bundle and configuration.

   Run the docker version command from the Docker client command shell. (download Docker for Mac or Docker for Windows on your laptop or install Docker on your linux client)

   $ docker version --format '{{println .Server.Platform.Name}}Client: {{.Client.Version}}{{range .Server.Components}}{{println}}{{.Name}}: {{.Version}}{{end}}'
   

   Example:

   

Prerequisite for installing vSphere CSI driver¶

1. Get cluster node information

$ kubectl get nodes -o wide

Example:

2. Make sure master node(s) are tainted, requires 2 taints as shown below (master= and uninitialized=true)

$ kubectl taint node 5bd6e49773ee-managers-1 node-role.kubernetes.io/master=:NoSchedule

$ kubectl taint node 5bd6e49773ee-managers-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

3. Make sure all the worker nodes are tainted with node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

$ kubectl taint node 5bd6e49773ee-workers-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

$ kubectl taint node 5bd6e49773ee-workers-2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

Verify the nodes are tainted

$ kubectl describe nodes | egrep "Taints:|Name:"

Example: