Docker Enterprise¶

Apply labels to a node¶

In this example, we’ll apply the ssd label to a node. Next, we’ll deploy a service with a deployment constraint to make sure the service is always scheduled to run on a node that has the ssd label.

1. Log in with administrator credentials in the MKE web interface.
2. Select Nodes in the left-hand navigation menu.
3. In the nodes list, select the node to which you want to apply labels.
4. In the details pane, select the edit node icon in the upper-right corner to edit the node.
5. In the Edit Node page, scroll down to the Labels section.
6. Select Add Label.
7. Add a label with the key disk and a value of ssd.
8. Click Save then dismiss the Edit Node page.
9. In the node’s details pane, select Labels to view the labels that are applied to the node.

You can also do this from the CLI by running:

docker node update --label-add <key>=<value> <node-id>

Deploy a service with constraints¶

When deploying a service, you can specify constraints, so that the service gets scheduled only on a node that has a label that fulfills all of the constraints you specify.

In this example, when users deploy a service, they can add a constraint for the service to be scheduled only on nodes that have SSD storage: node.labels.disk == ssd.

1. Navigate to the Stacks page.

2. Name the new stack “wordpress”.

3. Under Orchestrator Mode, select Swarm Services.

4. In the docker-compose.yml editor, paste the following stack file.

   version: "3.1"
   
   services:
     db:
       image: mysql:5.7
       deploy:
         placement:
           constraints:
             - node.labels.disk == ssd
         restart_policy:
           condition: on-failure
       networks:
         - wordpress-net
       environment:
         MYSQL_ROOT_PASSWORD: wordpress
         MYSQL_DATABASE: wordpress
         MYSQL_USER: wordpress
         MYSQL_PASSWORD: wordpress
     wordpress:
       depends_on:
         - db
       image: wordpress:latest
       deploy:
         replicas: 1
         placement:
           constraints:
             - node.labels.disk == ssd
         restart_policy:
           condition: on-failure
           max_attempts: 3
       networks:
         - wordpress-net
       ports:
         - "8000:80"
       environment:
         WORDPRESS_DB_HOST: db:3306
         WORDPRESS_DB_PASSWORD: wordpress
   
   networks:
     wordpress-net:
   

5. Click Create to deploy the stack, and when the stack deploys, click Done.

6. Navigate to the Nodes page, and click the node that has the disk label. In the details pane, click the Inspect Resource drop-down menu and select Containers.

7. Dismiss the filter and navigate to the Nodes page.

8. Click a node that doesn’t have the disk label. In the details pane, click the Inspect Resource drop-down menu and select Containers. There are no WordPress containers scheduled on the node. Dismiss the filter.

Add a constraint to a service by using the MKE web UI¶

You can declare the deployment constraints in your docker-compose.yml file or when you’re creating a stack. Also, you can apply them when you’re creating a service.

To check if a service has deployment constraints, navigate to the Services page and choose the service that you want to check. In the details pane, click Constraints to list the constraint labels.

To edit the constraints on the service, click Configure and select Details to open the Update Service page. Click Scheduling to view the constraints.

You can add or remove deployment constraints on this page.

Add new SANs to MKE¶

1. In the MKE web UI, log in with administrator credentials and navigate to the Nodes page.
2. Click on a manager node, and in the details pane, click Configure and select Details.
3. In the SANs section, click Add SAN, and enter one or more SANs for the cluster.
4. Once you’re done, click Save.

You will have to do this on every existsing manager node in the cluster, but once you have done so, the SANs are applied automatically to any new manager nodes that join the cluster.

You can also do this from the CLI by first running:

docker node inspect --format '{{ index .Spec.Labels "com.docker.ucp.SANs" }}' <node-id>
default-cs,127.0.0.1,172.17.0.1

This will get the current set of SANs for the given manager node. Append your desired SAN to this list, for example default-cs,127.0.0.1,172.17.0.1,example.com, and then run:

docker node update --label-add com.docker.ucp.SANs=<SANs-list> <node-id>

<SANs-list> is the list of SANs with your new SAN appended at the end. As in the web UI, you must do this for every manager node.

Default¶

• NamespaceLifecycle
• LimitRanger
• ServiceAccount
• PersistentVolumeLabel
• DefaultStorageClass
• DefaultTolerationSeconds
• NodeRestriction
• ResourceQuota
• PodNodeSelector
• PodSecurityPolicy

Custom¶

• UCPAuthorization:
  • Annotates Docker Compose-on-Kubernetes Stack resources with the identity of the user performing the request so that the Docker Compose-on-Kubernetes resource controller can manage Stacks with correct user authorization.
  • Detects when ServiceAccount resources are deleted so that they can be correctly removed from MKE’s Node scheduling authorization backend.
  • Simplifies creation of RoleBindings and ClusterRoleBindings resources by automatically converting user, organization, and team Subject names into their corresponding unique identifiers.
  • Prevents users from deleting the built-in cluster-admin ClusterRole or ClusterRoleBinding resources.
  • Prevents under-privileged users from creating or updating PersistintVolume resources with host paths.
  • Works in conjunction with the built-in PodSecurityPolicies admission controller to prevent under-privileged users from creating Pods with privileged options.
• CheckImageSigning: Enforces MKE’s Docker Content Trust policy which, if enabled, requires that all pods use container images which have been digitally signed by trusted and authorized users which are members of one or more teams in MKE.
• UCPNodeSelector: Adds a com.docker.ucp.orchestrator.kubernetes:* toleration to pods in the kube-system namespace and removes com.docker.ucp.orchestrator.kubernetes tolerations from pods in other namespaces. This ensures that user workloads do not run on swarm-only nodes, which MKE taints with com.docker.ucp.orchestrator.kubernetes:NoExecute. It also adds a node affinity to prevent pods from running on manager nodes depending on MKE’s settings.

Business metrics¶

These are high-level aggregate metrics that typically combine technical, financial, and organizational data to create metrics for business leaders of the IT infrastructure. Some examples of business metrics might be:

• Company or division-level application downtime
• Aggregate resource utilization
• Application resource demand growth

Application metrics¶

These are metrics about domain of APM tools like AppDynamics or DynaTrace and provide metrics about the state or performance of the application itself.

• Service state metrics
• Container platform metrics
• Host infrastructure metrics

Docker Enterprise 2.1 does not collect or expose application level metrics.

The following are metrics Docker Enterprise 2.1 collects, aggregates, and exposes:

Service state metrics¶

These are metrics about the state of services running on the container platform. These types of metrics have very low cardinality, meaning the values are typically from a small fixed set of possibilities, commonly binary.

• Application health
• Convergence of K8s deployments and Swarm services
• Cluster load by number of services or containers or pods

Web UI disk usage metrics, including free space, only reflect the Docker managed portion of the filesystem: /var/lib/docker. To monitor the total space available on each filesystem of a MKE worker or manager, you must deploy a third party monitoring solution to monitor the operating system.

Deploy Prometheus on worker nodes¶

MKE deploys Prometheus by default on the manager nodes to provide a built-in metrics backend. For cluster sizes over 100 nodes or for use cases where scraping metrics from the Prometheus instances are needed, we recommend that you deploy Prometheus on dedicated worker nodes in the cluster.

To deploy Prometheus on worker nodes in a cluster:

1. Begin by sourcing an admin bundle.

2. Verify that ucp-metrics pods are running on all managers.

   $ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide NAME
   READY     STATUS    RESTARTS   AGE       IP              NODE
   ucp-metrics-hvkr7   3/3       Running   0          4h
   192.168.80.66   3a724a-0
   

3. Add a Kubernetes node label to one or more workers. Here we add a label with key “ucp-metrics” and value “” to a node with name “3a724a-1”.

   $ kubectl label node 3a724a-1 ucp-metrics=
   node "test-3a724a-1" labeled
   

   SELinux Prometheus Deployment for MKE 3.1.0, 3.1.1, and 3.1.2

   If you are using SELinux, you must label your ucp-node-certs directories properly on your worker nodes before you move the ucp-metrics workload to them. To run ucp-metrics on a worker node, update the ucp-node-certs label by running sudo chcon -R system_u:object_r:container_file_t:s0 /var/lib/docker/volumes/ucp-node-certs/_data.

4. Patch the ucp-metrics DaemonSet’s nodeSelector using the same key and value used for the node label. This example shows the key “ucp-metrics” and the value “”.

   $ kubectl -n kube-system patch daemonset ucp-metrics --type json -p
   '[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value":
   {"ucp-metrics": ""}}]' daemonset "ucp-metrics" patched
   

5. Observe that ucp-metrics pods are running only on the labeled workers.

   $ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide NAME
   READY     STATUS        RESTARTS   AGE       IP              NODE
   ucp-metrics-88lzx   3/3       Running       0          12s
   192.168.83.1    3a724a-1 ucp-metrics-hvkr7   3/3       Terminating   0
   4h        192.168.80.66   3a724a-0
   

Configure external Prometheus to scrape metrics from MKE¶

To configure your external Prometheus server to scrape metrics from Prometheus in MKE:

1. Begin by sourcing an admin bundle.

2. Create a Kubernetes secret containing your bundle’s TLS material.

   (cd $DOCKER_CERT_PATH && kubectl create secret generic prometheus --from-file=ca.pem --from-file=cert.pem --from-file=key.pem)
   

3. Create a Prometheus deployment and ClusterIP service using YAML as follows.

   On AWS with Kube’s cloud provider configured, you can replace ClusterIP with LoadBalancer in the service YAML then access the service through the load balancer. If running Prometheus external to MKE, change the following domain for the inventory container in the Prometheus deployment from ucp-controller.kube-system.svc.cluster.local to an external domain to access MKE from the Prometheus node.

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: prometheus
   data:
     prometheus.yaml: |
       global:
         scrape_interval: 10s
       scrape_configs:
       - job_name: 'ucp'
         tls_config:
           ca_file: /bundle/ca.pem
           cert_file: /bundle/cert.pem
           key_file: /bundle/key.pem
           server_name: proxy.local
         scheme: https
         file_sd_configs:
         - files:
           - /inventory/inventory.json
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: prometheus
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: prometheus
     template:
       metadata:
         labels:
           app: prometheus
       spec:
         containers:
         - name: inventory
           image: alpine
           command: ["sh", "-c"]
           args:
           - apk add --no-cache curl &&
             while :; do
               curl -Ss --cacert /bundle/ca.pem --cert /bundle/cert.pem --key /bundle/key.pem --output /inventory/inventory.json https://ucp-controller.kube-system.svc.cluster.local/metricsdiscovery;
               sleep 15;
             done
           volumeMounts:
           - name: bundle
             mountPath: /bundle
           - name: inventory
             mountPath: /inventory
         - name: prometheus
           image: prom/prometheus
           command: ["/bin/prometheus"]
           args:
           - --config.file=/config/prometheus.yaml
           - --storage.tsdb.path=/prometheus
           - --web.console.libraries=/etc/prometheus/console_libraries
           - --web.console.templates=/etc/prometheus/consoles
           volumeMounts:
           - name: bundle
             mountPath: /bundle
           - name: config
             mountPath: /config
           - name: inventory
             mountPath: /inventory
         volumes:
         - name: bundle
           secret:
             secretName: prometheus
         - name: config
           configMap:
             name: prometheus
         - name: inventory
           emptyDir:
             medium: Memory
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: prometheus
   spec:
     ports:
     - port: 9090
       targetPort: 9090
     selector:
       app: prometheus
     sessionAffinity: ClientIP
   EOF
   

4. Determine the service ClusterIP.

   $ kubectl get service prometheus
   NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
   prometheus   ClusterIP   10.96.254.107   <none>        9090/TCP   1h
   

5. Forward port 9090 on the local host to the ClusterIP. The tunnel created does not need to be kept alive and is only intended to expose the Prometheus UI.

   ssh -L 9090:10.96.254.107:9090 ANY_NODE
   

6. Visit http://127.0.0.1:9090 to explore the MKE metrics being collected by Prometheus.

MKE metrics¶

The following table lists the metrics that MKE exposes in Prometheus, along with descriptions. Note that only the metrics labeled with ucp_ are documented. Other metrics are exposed in Prometheus but are not documented.

Creating roles¶

You create Kubernetes roles either through the CLI using kubectl or through the MKE web interface.

To create a Kubernetes role in the MKE web interface:

1. From the MKE UI, select Access Control.
2. From the left navigation menu, select Roles.
3. Select the Kubernetes tab at the top of the window.
4. Select Create to create a Kubernetes role object.
5. Select a namespace from the Namespace drop-down list. Selecting a specific namespace creates a role for use in that namespace, but selecting all namespaces creates a ClusterRole where you can create rules for cluster-scoped Kubernetes resources as well as namespaced resources.
6. Provide the YAML for the role, either by entering it in the Object YAML editor or select Click to upload a .yml file to choose and upload a .yml file instead.
7. When you have finished specifying the YAML, Select Create to complete role creation.

Creating role grants¶

Kubernetes provides two types of role grants:

• ClusterRoleBinding which applies to all namespaces
• RoleBinding which applies to a specific namespace

To create a grant for a Kubernetes role in the MKE web interface:

1. From the MKE UI, select Access Control.
2. From the left navigation menu, select Grants.
3. Select the Kubernetes tab at the top of the window. All grants to Kubernetes roles can be viewed in the Kubernetes tab.
4. Select Create New Grant to start the Create Role Binding wizard and create a new grant for a given user, team or service.
5. Select the subject type. Your choices are:
   • All Users
   • Organizations
   • Service account
6. To create a user role binding, select a username from the Users drop-down list then select Next.
7. Select a resource set for the subject. The default namespace is automatically selected. To use a different namespace, select the Select Namespace button next to the desired namespace. For Cluster Role Binding, slide the Apply Role Binding to all namespaces selector to the right.
8. Select Next to continue.
9. Select the Cluster Role from the drop-down list. If you create a ClusterRoleBinding (by selecting Apply Role Binding to all namespaces) then you may only select ClusterRoles. If you select a specific namespace, you can choose any role from that namespace or any ClusterRole.
10. Select Create to complete creating the grant.

Logging levels¶

To allow more control to administrators over the audit logging, three audit logging levels are provided:

• None: audit logging is disabled
• Metadata: includes the following:
  • Method and API endpoint for the request
  • MKE user who made the request
  • Response Status (success or failure)
  • Timestamp of the call
  • Object ID of any created or updated resource (for create or update API calls). We do not include names of created or updated resources
  • License Key
  • Remote Address
• Request: includes all fields from the Metadata level as well as the request payload.

Benefits¶

You can use audit logs to help with the following use cases:

• Historical troubleshooting - Audit logs are helpful in determining a sequence of past events that explain why an issue occurred.
• Security analysis and auditing - Security is one of the primary uses for audit logs. A full record of all user interactions with the container infrastructure gives your security team full visibility to questionable or attempted unauthorized accesses.
• Chargeback - You can use audit logs and information about the resources to generate chargeback information.
• Alerting - If there is a watch on an event stream or a notification created by the event, alerting features can be built on top of event tools that generate alerts for ops teams (PagerDuty, OpsGenie, Slack, or custom solutions).

Enabling MKE audit logging¶

MKE audit logging can be enabled via the MKE web user interface, the MKE API or via the MKE configuration file.

[audit_log_configuration]
  level = "metadata"
  support_dump_include_audit_logs = false

Accessing audit logs¶

The audit logs are exposed today through the ucp-controller logs. You can access these logs locally through the Docker CLI or through an external container logging solution, such as ELK.

$ docker logs ucp-controller --tail 1
{"audit":{"auditID":"f8ce4684-cb55-4c88-652c-d2ebd2e9365e","kind":"docker-swarm","level":"metadata","metadata":{"creationTimestamp":null},"requestReceivedTimestamp":"2019-01-30T17:21:45.316157Z","requestURI":"/metricsservice/query?query=(%20(sum%20by%20(instance)%20(ucp_engine_container_memory_usage_bytes%7Bmanager%3D%22true%22%7D))%20%2F%20(sum%20by%20(instance)%20(ucp_engine_memory_total_bytes%7Bmanager%3D%22true%22%7D))%20)%20*%20100\u0026time=2019-01-30T17%3A21%3A45.286Z","sourceIPs":["172.31.45.250:48516"],"stage":"RequestReceived","stageTimestamp":null,"timestamp":null,"user":{"extra":{"licenseKey":["FHy6u1SSg_U_Fbo24yYUmtbH-ixRlwrpEQpdO_ntmkoz"],"username":["admin"]},"uid":"4ec3c2fc-312b-4e66-bb4f-b64b8f0ee42a","username":"4ec3c2fc-312b-4e66-bb4f-b64b8f0ee42a"},"verb":"GET"},"level":"info","msg":"audit","time":"2019-01-30T17:21:45Z"}

Sample logs¶

Here is a sample audit log for a Kubernetes cluster.

{"audit"; {
      "metadata": {...},
      "level": "Metadata",
      "timestamp": "2018-08-07T22:10:35Z",
      "auditID": "7559d301-fa6b-4ad6-901c-b587fab75277",
      "stage": "RequestReceived",
      "requestURI": "/api/v1/namespaces/default/pods",
      "verb": "list",
      "user": {"username": "alice",...},
      "sourceIPs": ["127.0.0.1"],
      ...,
      "requestReceivedTimestamp": "2018-08-07T22:10:35.428850Z"}}

Here is a sample audit log for a Swarm cluster.

{"audit"; {
      "metadata": {...},
      "level": "Metadata",
      "timestamp": "2018-08-07T22:10:35Z",
      "auditID": "7559d301-94e7-4ad6-901c-b587fab31512",
      "stage": "RequestReceived",
      "requestURI": "/v1.30/configs/create",
      "verb": "post",
      "user": {"username": "alice",...},
      "sourceIPs": ["127.0.0.1"],
      ...,
      "requestReceivedTimestamp": "2018-08-07T22:10:35.428850Z"}}

API endpoints ignored¶

The following API endpoints are ignored since they are not considered security events and may create a large amount of log entries.

• /_ping
• /ca
• /auth
• /trustedregistryca
• /kubeauth
• /metrics
• /info
• /version*
• /debug
• /openid_keys
• /apidocs
• /kubernetesdocs
• /manage

API endpoint information redacted¶

Information for the following API endpoints is redacted from the audit logs for security purposes:

• /secrets/create (POST)
• /secrets/{id}/update (POST)
• /swarm/join (POST)
• /swarm/update (POST) -/auth/login (POST)
• Kubernetes secrete create/update endpoints

Configure identity provider integration¶

There are values your identity provider needs for successful integration with MKE, as follows. These values can vary between identity providers. Consult your identity provider documentation for instructions on providing these values as part of their integration process.

Configure the SAML integration¶

To enable SAML authentication:

1. Go to the MKE web interface.
2. Navigate to the Admin Settings.
3. Select Authentication & Authorization.
4. In the SAML Enabled section, select Yes to display the required settings. The settings are grouped by those needed by the identity provider server and by those needed by MKE as a SAML service provider.
5. In IdP Metadata URL enter the URL for the identity provider’s metadata.
6. If the metadata URL is publicly certified, you can leave Skip TLS Verification unchecked and Root Certificates Bundle blank, which is the default. Skipping TLS verification is not recommended in production environments. If the metadata URL cannot be certified by the default certificate authority store, you must provide the certificates from the identity provider in the Root Certificates Bundle field.
7. In MKE Host enter the URL that includes the IP address or domain of your MKE installation. The port number is optional. The current IP address or domain appears by default.
8. To customize the text of the sign-in button, enter your button text in the Customize Sign In Button Text field. The default text is ‘Sign in with SAML’.
9. The Service Provider Metadata URL and Assertion Consumer Service (ACS) URL appear in shaded boxes. Select the copy icon at the right side of each box to copy that URL to the clipboard for pasting in the identity provider workflow.
10. Select Save to complete the integration.

Security considerations¶

You can download a client bundle to access MKE. A client bundle is a group of certificates downloadable directly from MKE web interface that enables command line as well as API access to MKE. It lets you authorize a remote Docker engine to access specific user accounts managed in Docker Enterprise, absorbing all associated RBAC controls in the process. You can now execute docker swarm commands from your remote machine that take effect on the remote cluster. You can download the client bundle in the Admin Settings under My Profile.

Configure IdP¶

Service Provider metadata is available at https://<SP Host>/enzi/v0/saml/metadata after SAML is enabled. The metadata link is also labeled as entityID.

Enable SAML and configure MKE¶

After MKE sends an AuthnRequest to the IdP, the following Assertion is expected:

• Subject includes a NameID that is identified as the username for MKE. In AuthnRequest, NameIDFormat is set to urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified. This allows maximum compatibility for various Identity Providers.

  <saml2:Subject>
     <saml2:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">mobywhale</saml2:NameID>
     <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
        <saml2:SubjectConfirmationData NotOnOrAfter="2018-09-10T20:04:48.001Z" Recipient="https://18.237.224.122/enzi/v0/saml/acs"/>
     </saml2:SubjectConfirmation>
  </saml2:Subject>
  

• An optional Attribute named fullname is mapped to the Full Name field in the MKE account.

  Note

  MKE uses the value of the first occurrence of an Attribute with Name="fullname" as the Full Name.

  <saml2:Attribute Name="fullname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
     <saml2:AttributeValue
        xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">user.displayName
     </saml2:AttributeValue>
  </saml2:Attribute>
  

• An optional Attribute named member-of is linked to the MKE team. The values are set in the MKE interface.

  Note

  MKE uses all AttributeStatements and Attributes in the Assertion with Name="member-of".

  <saml2:Attribute Name="member-of" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
     <saml2:AttributeValue
        xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">groupName
     </saml2:AttributeValue>
  </saml2:Attribute>
  

• An optional Attribute with the name is-admin is used to identify if the user is an administrator.

  Note

  When there is an Attribute with the name is-admin, the user is an administrator. The content in the AttributeValue is ignored.

  <saml2:Attribute Name="is-admin" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
     <saml2:AttributeValue
        xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">value_doe_not_matter
     </saml2:AttributeValue>
  </saml2:Attribute>
  

Service Provider configuration¶

Enter the Identity Provider’s metadata URL to obtain its metadata. To access the URL, you may need to provide the CA certificate that can verify the remote server.

Link Group memberships with users¶

Use the ‘edit’ or ‘create’ team dialog to associate SAML group assertion with the MKE team to synchronize user team membership when the user logs in.

Supported identity providers¶

• Okta 3.2.0

Typical steps involved in SCIM integration:¶

1. Configure SCIM for MKE.
2. Configure SCIM authentication and access.
3. Specify user attributes.