MKE release notes¶

Components¶

What’s new¶

• MKE now tags all analytics reports with the user license ID when telemetry is enabled. It does not, though, collect any further identifying information. In line with this change, the MKE configuration no longer contains the anonymize_tracking setting, and the MKE web UI no longer includes the Make data anonymous toggle (MKE-8316).
• MKE no longer exposes the Interlock NGINX ServerNamesHashBucketSize setting. The setting was confusing users because MKE adaptively calculates the setting and overrides any manual input (MKE-8306).
• Improved MKE controller memory usage due to the high-load MKE database (FIELD-3540).
• Improved the MKE database query performance for role-based access control (RBAC) information (FIELD-3540).
• Added the authz_cache_timeout setting to the MKE configuration, which allows the caching of role-based access control (RBAC) information for non-Kubernetes MKE resource listing APIs. When enabled, this setting improves API performance and reduces the MKE database load. MKE does not enable the cache by default (FIELD-3540).
• FELIX_LOGSEVERITYSCREEN can now adhere to a greater number of MKE log verbosity levels resulting in less log content when users do not want debug or error information (FIELD-2673).

Bug fixes¶

• Fixed an issue wherein previously-rejected MKE component tasks caused MKE upgrades to fail (FIELD-4032).

Known issues¶

• Due to potential port conflicts between kubectl and NodePort, it may not be possible to use kubectl where a NodePort is established throughout the cluster (FIELD-3495).

  Workaround:

  Reconfigure the ephemeral port range on each container host to avoid overlapping ports:

  1. Create the file /etc/sysctl.d/kubelet_ephemeral_port.conf:

     net.ipv4.ip_local_port_range=35536 60999
     

  2. Load the change for the current boot:

     sudo sysctl -p /etc/sysctl.d/kubelet_ephemeral_port.conf
     

  3. Restart kubelet:

     docker restart ucp-kubelet
     

  Wherever possible, Mirantis recommends that you put the Kubernetes node that you plan to restart into drain status, which thereby migrates running pods to other nodes. In the event that the kubelet restart lasts longer than five minutes, this migration will minimize the potential impact on those services.

  Undertake any restart of the kubelet on a manager node with care, as this action will impact the services and API of any Kubernetes system pod that restarts concurrently, until the manager node kubelet operates normally.

  Note that this workaround may not be a viable option in a production environment, as restarting the kubelet can result in any of the following:

  • If the restart takes longer than five minutes, Kubernetes will stop all of the pods running on the node and attempt to start them on a different node.
  • Pod or service health checks can fail during the restart.
  • Kubernetes system metrics may fail or be inaccurately reported until the restart is complete.
  • If the restart takes too long or fails, Kubernetes may designate the node as unhealthy. This can result in Kubernetes removing the node from the orchestrating pods until it redesignates the node as healthy.

Components¶

What’s new¶

• MKE now gives users the option to send the log and a support bundle to Mirantis Support when an upgrade fails (MKE-8133).

Bug fixes¶

• Fixed an issue wherein the default Interlock NGINX proxy server_names_hash_bucket_size could not handle very long host names, sometimes causing existing services to become unreachable. server_names_hash_bucket_size is now fully adaptive within hard bounds. (MKE-8262).
• Fixed an issue wherein enabling HitlessServiceUpdate while a proxy update is in progress caused the proxy update to stop (FIELD-3623).
• Fixed an issue wherein two files remained in the ucp-backup volume (/var/lib/docker/volumes/ucp-backup) after the completion of the back-up process. Now, following back-up, only the back-up archive and log file (if included) remain (FIELD-3612).
• Fixed an issue wherein users could not change new swarm configurations to use a non-default collection (FIELD-2297).
• Fixed an issue wherein MKE erroneously reported disconnected for drained nodes (FIELD-3771).

Components¶

What’s new¶

• Added the ability to use the CLI to send a support dump to Mirantis Customer Support, by including the --submit option with the support command (MKE-8150).

  Learn more

  • get-support
  • mirantis/ucp support

• Compose-on-Kubernetes will be deprecated in a future release (ENGDOCS-959).

• The LDAP search initiates stricter checks, and as such user syncing errors can no longer cause MKE users to be deactivated. User syncing now aborts when any of the following conditions are met:

  • An incorrect LDAP configuration is found
  • A configured LDAP URL is inaccessible
  • An LDAP URL that SearchResultReference points to is inaccessible

  (FIELD-3619).

Components¶

Bug fixes¶

• Fixed an issue with running Kubernetes on Azure wherein pods failed to start with the following error:

  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to
  set up sandbox container "[…]" network for pod "[…]": networkPlugin cni
  failed to set up pod "[…]" network: Failed to allocate address: Invalid
  address space
  

  FIELD-3635

Security¶

• Resolved an important security issue in Go’s encoding/xml package that affects all prior versions of MKE 3.2. Specifically, maliciously crafted XML markup was able to potentially mutate during round trips through Go’s decoder and encoder implementations.

  Implementations of Go-based SAML (Security Assertion Markup Language, an XML-based standard approach to Single Sign-On – SSO – on the web) are often vulnerable to tampering by an attacker injecting malicious markup to a correctly-signed SAML message. MKE uses crewjam/saml, a Go SAML implementation that is affected by the vulnerability and which is tracked by CVE-2020-27846.

  MKE-8149

Components¶

Bug fixes¶

• Fixed an issue wherein the enabling of HitlessServiceUpdate caused Interlock Proxy to fail, due to a lack of synchronization between the proxy config and secrets. The fix addressed secrets handling, and as a result the proxy no longer restarts when secrets are added, removed, or changed. (FIELD-2896).
• Performing a manual upgrade now removes older worker agents automatically, provided that all nodes have been upgraded (MKE-7993).
• Fixed an issue wherein clicking Admin Settings in the UI loaded a blank page after upgrade (FIELD-2293).
• Fixed an issue wherein selecting a particular network policy under Kubernetes > Configurations caused the UI to become blank (MKE-7959).
• Fixed a UI issue wherein a checkbox in Admin Settings > Scheduler reverted to an unchecked state following selection (MKE-8011).

Known issues¶

• It may not be possible to use kubectl where a NodePort has already been established throughout the cluster, due to potential port conflicts between kubectl and NodePort (FIELD-3495).

  Workaround:

  Restart the kubelet to resolve the port conflict, after which you can exec into the node.

  Wherever possible, it is recommended that you put the Kubernetes node that you plan to restart into drain status, thereby migrating running pods to other nodes. In the event that the kubelet restart lasts longer than five minutes, this migration will minimize the potential impact on those services.

  Restarting the kubelet on a manager node should be undertaken with care. The services and API of any Kubernetes system pod that restarts concurrently will be impacted until the manager node’s kubelet is operating normally.

  Note that this workaround may not be a viable option in a production environment, and that restarting the kubelet can result in any of the following:

  • If the restart takes longer than five minutes, Kubernetes will stop all of the pods running on the node and attempt to start them on a different node.
  • Pod or service health checks can fail during the restart.
  • Kubernetes system metrics may fail or be inaccurately reported until the restart is complete.
  • If the restart takes too long or fails, Kubernetes may designate the node as unhealthy. This can result in Kubernetes removing the node from the orchestrating pods until Kubernetes redesignates the node as healthy.

Components¶

Bug fixes¶

• Fixed various links to knowledge base articles in the UI (FIELD-3302).

Components¶

What’s new¶

• Added the allow_repos MKE configuration, to allow user-specified repos to bypass content trust check (useful for those who want to run dtr backup or dtr upgrade with content trust enabled) (FIELD-1710).

Bug fixes¶

• Fixed an UI issue that resulted in the display of a blank Admin Settings page whenever Docker content trust is not enabled (ENGORC-2914).

Security¶

• Upgraded Golang to 1.15.2 (ENGORC-7900).

Components¶

What’s new¶

• On Docker Hub, MKE images are now released to ‘mirantis’ instead of ‘docker’.

• We updated the location of our offline bundles for MKE from https://packages.docker.com/caas/ to https://packages.mirantis.com/caas/ for the following versions of MKE.

  • MKE 3.3.2
  • MKE 3.2.8
  • MKE 3.1.15

  Offline bundles for other previous versions of MKE will remain on the docker domain.

• Whitelisting of all MKE repos (FIELD-2723).

• Added tracing to Interlock (ENGORC-7565).

Bug fixes¶

• We fixed an issue in which Docker Content Trust was randomly failing to verify valid signatures (FIELD-2302).

• The MKE upgrade GUI create a command string that uses docker image pull docker/ucp:..... You should change it to `` docker image pull mirantis/ucp:….” for starting with MKE version 3.1.15 (ENGORC-7806).

• We fixed an issue that caused the following ucp-kubelet error when the docker root location (/var/lib/docker) was modified (ENGORC-7671).

  failed to load Kubelet config file
  /var/lib/docker/volumes/ucp-node-certs/_data/kubelet_daemon.conf,
  error failed to read kubelet config file
  "/var/lib/docker/volumes/ucp-node-certs/_data/kubelet_daemon.conf",
  error: open /var/lib/docker/volumes/ucp-node-certs/_data/kubelet_daemon.
  conf: no such file or directory
  

• We updated the container/ps APIs to require admin access (ENGORC-7618).

• We fixed an issue that prevented users from logging into MKE using Security Assertion Markup Language (SAML) after the root certificate for Active Directory Federation Services (ADFS) has been renewed (ENGORC-7754).

• We added support for installing MKE on cloud providers using cloud-provider=external (ENGORC-7686).

• We fixed an issue that allowed users unlimited login attempts in MKE, MSR, and eNZi (ENGORC-7742).

• We fixed an issue that prevented the HNS network from starting before starting the kube-proxy on Windows, which prevented kube bringup on the node (ENGORC-7961).

• We fixed an issue with the MKE user interface for Kubernetes pods that made it look like no data was returned if no vulnerabilities were found, instead of indicating a clean report (ENGORC-7685).

• We fixed an issue that caused Kubernetes windows nodes take too long to come up (ENGORC-7660).

• Added interlock configuration validation (ENGORC-7643).

• When HitlessServiceUpdate is enabled, the config service no longer waits for the proxy service to complete an update, thus reducing the delay between a configuration change being made and taking effect (FIELD-2152).

• Improved the speed of interlock API calls (ENGORC-7366).

• We fixed an issue that causes API path traversal (ENGORC-7744).

• Using Docker Enterprise with the AWS Kubernetes cloud provider requires the metadata service for Linux nodes. Enabling the metadata service also enables access from Linux workload containers. It’s a best practice to limit access to Linux workload containers. You can create an iptable to block access to workload containers. It can be made persistent by adding it to the docker systemd unit (ENGORC-7620).

  • Create a file /etc/systemd/system/docker.service.d/block-aws-metadata.conf with the following contents:

    # /etc/systemd/system/docker.service.d/block-aws-metadata.conf
    [Service]
    ExecStartPost=/bin/sh -c ""iptables -I DOCKER-USER -d 169.254.169.254/32 -j DROP
    

  • Reload the systemd configuration (systemctl daemon-reload).

    The iptables rule will now be installed every time the Docker engine starts.

  • Check for the presence of the rule with iptables -nvL DOCKER-USER.

• We fixed an issue in which the MKE support dump script checks for the obsolete legacy DTR (1.x) dtr-br bridge network, and being unable to find it subsequently reports an error in dsinfo.txt (FIELD-2670).

• Fixed an issue wherein swarm rotated the CA causing AuthorizeNode to fail (FIELD-2875).

Security¶

• We updated our Go engine to address CVE-2020-14040 (ENGORC-7772)
• We fixed an issue that allowed users unlimited login attempts in MKE, MSR, and eNZi.
• We fixed an issue that caused the “docker ps” command to provide the incorrect status (starting) for running containers after sourcing a client bundle. This command now shows the correct (healthy) status value (ENGORC-7721).
• We fixed an issue that allowed unpriviledged user account to access plain text data from backups, including encrypted backups, such as user password hashes, eNZi signing keys, and the Kubernetes service account key, which may enable direct compromise of the Docker Enterprise cluster (ENGORC-7631).
• We fixed an issue that allowed access to containers running in other collections in order to escalate their privileges throughout the cluster (ENGORC-7595).
• Fixed an issue that causes API path traversal (ENGORC-7744).

Bug Fixes¶

• The stack’s specific configuration fails to display details in the MKE UI (Swarm -> Configurations -> ..choose a config..) whenever configurations are created that include the label com.docker.stack.namespace (occurs when the new config is created in the compose file deployed with Docker stack using client-bundle). The config’s details can be seen, though, via ucp-bundle and also with a direct API call. (ENGORC-7486)
• When leader change occurs in swarmkit the new leader’s node address can change to 0.0.0.0. The ucp-metrics inventory.json file may adopt a 0.0.0.0 target address as a result, thus producing a situation wherein the MKE dashboard is unable to display metrics for the leader node. (ENGORC-3256)
• Adding in-house certificates to MKE can result in a reconciler loop when the cert’s issuer CN is empty. (ENGORC-3255)
• Any LDAP search that returns 0 members (a normal result) results in the aborting of the entire LDAP sync. (ENGORC-3237)

Components¶

Security¶

• Upgraded Golang to 1.13.8.
• Updated several Golang vendors to address security issues.
• Fixed an issue that caused file descriptors to remain open in ucp-agent.

Bug Fixes¶

• Any LDAP search that returns 0 members (a normal result) results in the aborting of the entire LDAP sync. (ENGORC-3237)
• When leader change occurs in swarmkit the new leader’s node address can change to 0.0.0.0. The ucp-metrics inventory.json file may adopt a 0.0.0.0 target address as a result, thus producing a situation wherein the MKE dashboard is unable to display metrics for the leader node. (ENGORC-3256)
• Adding in-house certificates to MKE can result in a reconciler loop when the cert’s issuer CN is empty. (ENGORC-3255)
• The stack’s specific configuration fails to display details in the MKE UI (Swarm -> Configurations -> ..choose a config..) whenever configurations are created that include the label com.docker.stack.namespace (occurs when the new config is created in the compose file deployed with Docker stack using client-bundle). The config’s details can be seen, though, via ucp-bundle and also with a direct API call. (ENGORC-7486)
• Updated swarm-rafttool. (FIELD-2081)
• Fixed a permission issue to prevent reconciler from looping. (FIELD-2235)
• Improved the speed to generate a Windows support dump. (FIELD-2304)

Components¶

Known issues¶

• MKE currently turns on vulnerability information for images deployed within MKE by default for upgrades. This may cause clusters to fail due to performance issues. (ENGORC-2746)

• For Red Hat Enterprise Linux (RHEL) 8, if firewalld is running and FirewallBackend=nftables is set in /etc/firewalld/firewalld.conf, change this to FirewallBackend=iptables, or you can explicitly run the following commands to allow traffic to enter the default bridge (docker0) network:

  firewall-cmd --permanent --zone=trusted --add-interface=docker0
  firewall-cmd --reload
  

Kubernetes¶

• Enabled support for a user-managed Kubernetes KMS plugin.

Components¶

Known issues¶

• MKE currently turns on vulnerability information for images deployed within MKE by default for upgrades. This may cause clusters to fail due to performance issues. (ENGORC-2746)

• For Red Hat Enterprise Linux (RHEL) 8, if firewalld is running and FirewallBackend=nftables is set in /etc/firewalld/firewalld.conf, change this to FirewallBackend=iptables, or you can explicitly run the following commands to allow traffic to enter the default bridge (docker0) network:

  firewall-cmd --permanent --zone=trusted --add-interface=docker0
  firewall-cmd --reload
  

Platforms¶

• RHEL 8.0 is now supported.

Kubernetes¶

• Kubernetes has been upgraded to version 1.14.8 that fixes CVE-2019-11253.
• Added a feature that allows the user to enable SecureOverlay as an add-on on MKE via an install flag called secure-overlay. This flag enables IPSec Network Encryption in Kubernetes.

Security¶

• Upgraded Golang to 1.12.12. (ENGORC-2762)
• Fixed an issue that allowed a user with a Restricted Control role to obtain Admin access to MKE. (ENGORC-2781)

Bug fixes¶

• Fixed an issue where MKE 3.2 backup performs an append not overwrite when --file switch is used. (FIELD-2043)
• Fixed an issue where the Calico/latest image was missing from the MKE offline bundle. (FIELD-1584)
• Image scan result aggregation is now disabled by default for new MKE installations. This feature can be configured by a new ImageScanAggregationEnabled setting in the MKE tuning config. (ENGORC-2746)
• Adds authorization checks for the volumes referenced by the VolumesFrom Containers option. Previously, this field was ignored by the container create request parser, leading to a gap in permissions checks. (ENGORC-2781)

Components¶

UI¶

• Fixes a UI issue that caused incorrect line breaks at pre-logon banner notification (ENGORC-2678)
• Users have an option to store sessionToken per window tab session. (ENGORC-2597)

Kubernetes¶

• Kubernetes has been upgraded to version 1.14.7. For more information, see the Kubernetes Release Notes.
• Enabled Kubernetes Node Authorizer Plugin. (ENGORC-2652)

Networking¶

• Interlock has been upgraded to version 3.0.0. This upgrade includes the following updates:
  • New Interlock configuration options:
    • HitlessServiceUpdate: When set to true, the proxy service no longer needs to restart when services are updated, reducing service interruptions. The proxy also does not have to restart when services are added or removed, as long as the set of service networks attached to the proxy is unchanged. If secrets or service networks need to be added or removed, the proxy service will restart as in previous releases. (ENGCORE-792)
    • Networks: Defines a list of networks to which the proxy service will connect at startup. The proxy service will only connect to these networks and will no longer automatically connect to back-end service networks. This allows administrators to control which networks are used to connect to the proxy service and to avoid unnecessary proxy restarts caused by network changes . (ENGCORE-912)
  • Log an error if the com.docker.lb.network label does not match any of the networks to which the service is attached. (ENGCORE-837)
  • Do not generate an invalid NGINX configuration file if HTTPVersion is invalid. (FIELD-2046)

Bug fixes¶

• Upgraded RethinkDB Go Client to v5. (ENGORC-2704)
• Fixes an issue that caused slow response with increasing number of collections. (ENGORC-2638)

Components¶

Bug fixes¶

• Fixes an issue where MKE did not install on GCP due to missing metadata.google.internal in /etc/hosts

Kubernetes¶

• Kubernetes has been upgraded to version 1.14.6. For more information, see the Kubernetes Release Notes.
• Kubernetes DNS has been upgraded to 1.14.13 and is now deployed with more than one replica by default.

Networking¶

• Calico has been upgraded to version 3.8.2. For more information, see the Calico Release Notes.
• Interlock has been upgraded to version 2.6.1.
• The azure-ip-count variable is now exposed at install time, allowing a user to customize the number of IP addresses MKE provisions for each node.

Security¶

• Upgraded Golang to 1.12.9.
• Added CSP header to prevent cross-site scripting attacks (XSS)

Bootstrap¶

• Fixes various issues in install, uninstall, backup, and restore when MKE Telemetry data had been disabled. (ENGORC-2593)

Components¶

Security¶

Refer to MKE image vulnerabilities for details regarding actions to be taken, timeline, and any status updates, issues, and recommendations.

New features¶

• Group Managed Service Accounts (gMSA). On Windows, you can create or update a service using --credential-spec with the config://<config-name> format. This passes the gMSA credentials file directly to nodes before a container starts.
• Open Security Controls Assessment Language (OSCAL). OSCAL API endpoints have been added in MKE and MCR. These endpoints are enabled by default.
• Container storage interface (CSI). Version 1.0 of the CSI specification is now supported for container orchestrators to manage storage plugins. Note: As of May 2019, none of the available CSI drivers are production quality and are considered pre-GA.
• Internet Small Computer System Interface (iSCSI). Using iSCSI, a storage admin can now provision a MKE cluster with persistent storage from which MKE end users can request storage resources without needing underlying infrastructure knowledge.
• System for Cross-domain Identity Management (SCIM). SCIM implementation allows proactive synchronization with MKE and eliminates manual intervention for changing user status and group membership.
• Support for Pod Security Policies (PSPs) within Kubernetes. Pod Security Policies are enabled by default in MKE 3.2 allowing platform operators to enforce security controls on what can run on top of Kubernetes.
• Client Cert-based Authentication
  • Users can now use MKE client bundles for MSR authentication.
  • Users can now add their client certificate and key to their local MCR for performing pushes and pulls without logging in.
  • Users can now use client certificates to make API requests to MSR instead of providing their credentials.

Enhancements¶

Deprecations¶

The following features are deprecated in MKE 3.2:

docker node update --label-add com.docker.ucp.agent-pause=true <NODE>

Kubernetes¶

• Integrated Kubernetes Ingress
• You can now dynamically deploy L7 routes for applications, scale out multi-tenant ingress for shared clusters, and give applications TLS termination, path-based routing, and high-performance L7 load-balancing in a centralized and controlled manner.
• Kubernetes has been upgraded to version 1.14.3. For more information, see the Kubernetes Release Notes.

Known issues¶

Operation for "\"kubernetes.io/local-volume/local-pxjz5\"" failed. No retries
permitted until 2019-07-18 20:28:28.745186772 +0000 UTC m=+5936.009498175
(durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"local-pxjz5\"
(UniqueName: \"kubernetes.io/local-volume/local-pxjz5\") pod
\"pod-subpath-test-local-preprovisionedpv-l7k9\" (UID: \"364a339d-a98d-11e9-8d2d-0242ac11000b\")
: local: failed to mount device /dev/loop0 at
/var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pxjz5 (fstype: ),
error exit status 32"

[366633.029514] EXT4-fs (loop3): Couldn't mount RDWR because of SUSE-unsupported optional feature METADATA_CSUM. Load module with allow_unsupported=1.

[fs_types]...
ext4 = {features = has_journal,extent,huge_file,flex_bg,metadata_csum,64bit,dir_nlink,extra_isizeinode_size = 256}

docker service create --mode=global --restart-condition none --mount
type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock mavenugo/swarm-exec:17.03.0-ce docker
exec ucp-kubelet "/bin/bash" "-c" "sed -i 's/metadata_csum,//g' /etc/mke2fs.conf"

/sbin/iptables --wait -I FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables --wait -I FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables --wait -I  POSTROUTING -s 172.17.0.0/24 ! -o docker0 -j MASQUERADE

time="2019-05-22T00:27:54Z" level=fatal msg="the following required ports are blocked on your host: 179, 443, 2376, 6443, 6444, 10250, 12376, 12378 - 12386.  Check your firewall settings"

$ sudo yum downgrade container-selinux-2.74-1.el7

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    subjectName: kube-system-persistent-volume-binder
  name: kube-system-persistent-volume-binder:cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: persistent-volume-binder
  namespace: kube-system

{kubelet ip-172-31-13-214.us-west-2.compute.internal} FailedMount: MountVolume.WaitForAttach failed for volume "iscsi-4mpvj" : exit   status 127"

docker service create --mode=global --restart-condition none --mount   type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock mavenugo/swarm-exec:17.03.0-ce docker exec ucp-kubelet "/bin/bash" "-c" "echo /rootfs/usr/lib64 >> /etc/ld.so.conf.d/libc.conf && echo /rootfs/lib64 >> /etc/ld.so.conf.d/libc.conf && ldconfig"
4b1qxigqht0vf5y4rtplhygj8
overall progress: 0 out of 3 tasks
overall progress: 0 out of 3 tasks
overall progress: 0 out of 3 tasks
ugb24g32knzv: running
overall progress: 0 out of 3 tasks
overall progress: 0 out of 3 tasks
overall progress: 0 out of 3 tasks
overall progress: 0 out of 3 tasks

<Ctrl-C>
Operation continuing in background.
Use `docker service ps 4b1qxigqht0vf5y4rtplhygj8` to check progress.

$ docker service ps 4b1qxigqht0vf5y4rtplhygj8
ID                  NAME                                         IMAGE                            NODE                                DESIRED STATE       CURRENT STATE
ERROR               PORTS
bkgqsbsffsvp        hopeful_margulis.ckh79t5dot7pdv2jsl3gs9ifa   mavenugo/swarm-exec:17.03.0-ce   user-testkit-4DA6F6-sles-1   Shutdown            Complete 7 minutes ago
nwnur7r1mq77        hopeful_margulis.2gzhtgazyt3hyjmffq8f2vro4   mavenugo/swarm-exec:17.03.0-ce   user-testkit-4DA6F6-sles-0   Shutdown            Complete 7 minutes ago
uxd7uxde21gx        hopeful_margulis.ugb24g32knzvvjq9d82jbuba1   mavenugo/swarm-exec:17.03.0-ce   user
-testkit-4DA6F6-sles-2   Shutdown            Complete 7 minutes ago

Components¶