Mirantis Kubernetes Engine release notes

Mirantis Kubernetes Engine release notes

UCP is now MKE

The product formerly known as Univeral Control Plane (UCP) is now Mirantis Kubernetes Engine (MKE).

This document describes the latest changes, additions, known issues, and fixes for Mirantis Kubernetes Engine version 3.3. You can find the release notes for other supported versions here:

Version 3.3

3.3.5

(2020-12-17)

Components

Component

Version

MKE

3.3.5

Kubernetes

1.18.10

Calico

3.14.1

Calico for Windows

3.12.1

Interlock

3.2.0

Interlock NGINX proxy

1.17.10

Istio Ingress

1.4.10

CoreDNS

1.7.0

etcd

3.4.3

CSI Attacher

2.1.1

CSI Provisioner

1.4.0

CSI Snapshotter

1.2.2

CSI Resizer

0.4.0

CSI Node Driver Registrar

1.2.0

CSI Liveness Probe

1.1.0

What’s new

  • At the start of the upgrade process MKE now performs two checks, and if either are not met, the upgrade will not proceed (MKE-8000).

    • A backup of MKE has taken place within the past hour (to override this check, include --force-recent-backup flag in the upgrade command).

    • Minimum system requirements for both manager nodes and worker nodes are in place (to override this check, include the --force-minimums flag in the upgrade command).

      • manager nodes: 16GB RAM, 4 virtual CPUs, 25GB storage

      • worker nodes: 4GB RAM, 2 virtual CPUs, 25GB storage

  • Updated to React 17.

Bug fixes

  • Fixed an issue with hiding the Swarm UI to provide a Kube-only view (MKE-7901).

  • Fixed various links to knowledge base articles in the UI (FIELD-3302).

  • Fixed an issue that caused docker pull to fail under DCT when registry addr contained a port number (FIELD-3308).

  • Fixed a performance issue caused by excessive polling in the UI (MKE-8019).

  • Fixed a display issue in the team UI that occurred when LDAP or SAML were enabled (MKE-8018).

3.3.4

(2020-11-12)

Components

Component

Version

MKE

3.3.4

Kubernetes

1.18.10

Calico

3.14.1

Calico for Windows

3.12.1

Interlock

3.2.0

Interlock NGINX proxy

1.17.10

Istio Ingress

1.4.10

CoreDNS

1.7.0

etcd

3.4.3

CSI Attacher

2.1.1

CSI Provisioner

1.4.0

CSI Snapshotter

1.2.2

CSI Resizer

0.4.0

CSI Node Driver Registrar

1.2.0

CSI Liveness Probe

1.1.0

What’s new

  • Updated our version of Kubernetes to 1.18.10 (ENGORC-7894).

  • Upgraded Golang to 1.15.2 (ENGORC-7900).

  • Warnings have been added to the bootstrapper installaion process to indicate when the configured Kubernetes Pod/Service CIDR ranges intersect with the default Docker/Swarm network address pools (ENGORC-7776, FIELD-2761).

Bug fixes

  • Fixed a network communications issue with Kubernetes on Windows workloads that only affected Windows nodes with MKE 3.3.3 installed. Following an upgrade to MKE 3.3.4, the issue can be cleared by removing the ucp-bridge network from affected Windows nodes:

    docker network rm ucp-bridge
    

    (ENGORC-7930)

  • Fixed an issue wherein the MKE agent did not reconcile the etcd certificate secret tied to MKE node certificates (for Kubernetes compose API deployment) upon renewal of the certificates following their default three-month expiration (ENGORC-7849, FIELD-2323).

  • Fixed an issue wherein the You are currently using the latest version of MKE. message would display even when there is an available upgrade (ENGORC-7807).

  • Fixed an issue wherein the MKE upgrade GUI referenced the docker/ucp package location and not the mirantis/ucp package location (ENGORC-7806).

  • Fixed an UI issue that resulted in the display of a blank Admin Settings page whenever Docker content trust is not enabled (ENGORC-2914).

3.3.3

(2020-09-15)

Components

Component

Version

MKE

3.3.3

Kubernetes

1.18.8

Calico

3.14.1

Calico for Windows

3.12.1

Interlock

3.2.0

Interlock NGINX proxy

1.17.10

Istio Ingress

1.4.10

CoreDNS

1.7.0

etcd

3.4.3

CSI Attacher

2.1.1

CSI Provisioner

1.4.0

CSI Snapshotter

1.2.2

CSI Resizer

0.4.0

CSI Node Driver Registrar

1.2.0

CSI Liveness Probe

1.1.0

What’s new

  • MKE now supports Mirantis Container Cloud. For more information, refer to Mirantis Container Cloud documentation.

  • Updated our version of Kubernetes to 1.18.8 (ENGORC-7822).

Bug fixes

  • We fixed an issue that caused SAML ACS and Metadata URL endpoints to redirect to a missing /enzi/v0 (ENGORC-7842).

  • We fixed an issue whereby SAML Authbenication Endpoint could be used to forcibly log users out (ENGORC-7739).

  • We fixed an issue whereby Interlock failed to enforce the MKE RBAC on the Interlock certificate and key file lables com.docker.lb.ssl_key and com.docker.lb.ssl_cert (ENGORC-7741).

  • We fixed an issue whereby MKE component containers were continuing to use the default Docker bridge network (ENGORCS-7617).

  • We added logic to allow the UI to surface the form field validation errors, thus fixing an issue in which the Auth Login settings were producing unactionable errors (ENGORC-7352).

Known issues

  • Whenever 100 or more Windows nodes are running in Kubernetes mode, within 48 hours a small portion of those nodes may become unavailable with heartbeat failure or kubelet is unhealthy error messages.

    Workaround: Run docker swarm leave and then docker swarm join <swarm-token> on the affected Windows nodes.

    Mirantis recommends that you refrain from upgrading to 3.3.3 if you are running Kubernetes deployments on Windows at the aforementioned scale.

    MKE clusters running only Linux nodes are unaffected.

3.3.2

(2020-08-10)

Components

Component

Version

MKE

3.3.2

Kubernetes

1.17.9

Calico

3.12.2

Calico for Windows

3.12.1

Interlock

3.2.0

Interlock NGINX proxy

1.17.10

Istio Ingress

1.4.10

What’s new

  • On Docker Hub, MKE images are now released to ‘mirantis’ instead of ‘docker’.

  • We updated the location of our offline bundles for MKE from https://packages.docker.com/caas/ to https://packages.mirantis.com/caas/ for the following versions of MKE.

    • MKE 3.3.2

    • MKE 3.2.8

    • MKE 3.1.15

    Offline bundles for other previous versions of MKE will remain on the docker domain. You can see the installation links for all supported bundles in the MKE installation instructions.

  • We updated our version of Kubernetes to 1.17.9.

  • We updated our version of Istio to 1.4.10.

  • Whitelisting of all MKE repos (FIELD-2723).

  • Added tracing to Interlock (ENGORC-7565).

Bug fixes

  • We fixed an issue in which Docker Content Trust was randomly failing to verify valid signatures (FIELD-2302).

  • We fixed an issue in which ucp:3.3.2-tp1 images –list commands cause Cannot connect to the Docker daemon errors (ENGORC-7774).

  • The MKE upgrade GUI create a command string that uses docker image pull mirantis/ucp:..... You should change it to `` docker image pull mirantis/ucp:….”`` (ENGORC-7806).

  • We fixed an issue that caused the following ucp-kubelet error when the docker root location (/var/lib/docker) was modified (ENGORC-7671).

    failed to load Kubelet config file
    /var/lib/docker/volumes/ucp-node-certs/_data/kubelet_daemon.conf,
    error failed to read kubelet config file
    "/var/lib/docker/volumes/ucp-node-certs/_data/kubelet_daemon.conf",
    error: open /var/lib/docker/volumes/ucp-node-certs/_data/kubelet_daemon.
    conf: no such file or directory
    
  • We fixed an issue that prevented setting istio-policy and istio-telemetry affinity rules on Linux nodes (ENGORC-7682).

  • We updated the container/ps APIs to require admin access (ENGORC-7618).

  • We fixed an issue that prevented users from logging into MKE using Security Assertion Markup Language (SAML) after the root certificate for Active Directory Federation Services (ADFS) has been renewed (ENGORC-7754).

  • We added support for installing MKE on cloud providers using cloud-provider=external (ENGORC-7686).

  • We added a banner to the Istio Ingress configuration page in MKE to encourage users to deploy the full Istio service mesh and related UI technologies from upstream (ENGORC-7755).

  • We fixed an issue that allowed users unlimited login attempts in MSR, MKE, and eNZi (ENGORC-7742).

  • We fixed an issue that prevented the HNS network from starting before starting the kube-proxy on Windows, which prevented kube bringup on the node (ENGORC-7961).

  • We fixed an issue with the MKE user interface for Kubernetes pods that made it look like no data was returned if no vulnerabilities were found, instead of indicating a clean report (ENGORC-7685).

  • We added warning about affinity and taint rules for Istio Ingress configuration (ENGORC-7684).

  • We fixed an issue that caused Kubernetes windows nodes take too long to come up (ENGORC-7660).

  • Added interlock configuration validation (ENGORC-7643).

  • When HitlessServiceUpdate is enabled, the config service no longer waits for the proxy service to complete an update, thus reducing the delay between a configuration change being made and taking effect (FIELD-2152).

  • Improved the speed of interlock API calls (ENGORC-7366).

  • We fixed an issue that causes API path traversal (ENGORC-7744).

  • Using MKE with the AWS Kubernetes cloud provider requires the metadata service for Linux nodes. Enabling the metadata service also enables access from Linux workload containers. It’s a best practice to limit access to Linux workload containers. You can create an iptable to block access to workload containers. It can be made persistent by adding it to the docker systemd unit (ENGORC-7620).

    • Create a file /etc/systemd/system/docker.service.d/block-aws-metadata.conf with the following contents:

      # /etc/systemd/system/docker.service.d/block-aws-metadata.conf
      [Service]
      ExecStartPost=/bin/sh -c ""iptables -I DOCKER-USER -d 169.254.169.254/32 -j DROP
      
    • Reload the systemd configuration (systemctl daemon-reload).

      The iptables rule will now be installed every time MCR starts.

    • Check for the presence of the rule with iptables -nvL DOCKER-USER.

  • We fixed an issue in which the MKE support dump script checks for the obsolete legacy MSR (1.x) dtr-br bridge network, and being unable to find it subsequently reports an error in dsinfo.txt (FIELD-2670).

  • We fixed an issue in which the Windows node stays in “kubelet not ready” state (ENGORC-7566).

  • We fixed an issue in which all node/edit pages rendered blank when a user logs in as MKE Admin, as a result of attempts to set Kubernetes orchestration mode) (ENGORC-2819).

Security

  • We updated our Go engine to address CVE-2020-14040 (ENGORC-7772)

  • We fixed an issue that allowed users unlimited login attempts in MSR, MKE, and eNZi.

  • We fixed an issue that prevented Istio from reporting usage data (ENGORC-7738).

  • We fixed an issue that caused the “docker ps” command to provide the incorrect status (starting) for running containers after sourcing a client bundle. This command now shows the correct (healthy) status value (ENGORC-7721).

  • We fixed an issue that allowed unpriviledged user account to access plain text data from backups, including encrypted backups, such as user password hashes, eNZi signing keys, and the Kubernetes service account key, which may enable direct compromise of the MKE cluster (ENGORC-7631).

  • We fixed an issue that allowed access to containers running in other collections in order to escalate their privileges throughout the cluster (ENGORC-7595).

  • Fixed an issue that causes API path traversal (ENGORC-7744).

3.3.1

(2020-06-24)

Components

Component

Version

MKE

3.3.1

Kubernetes

1.17.6

Calico

3.12.2

Calico for Windows

3.12.1

Interlock

3.1.3

Interlock NGINX proxy

1.17.10

Bug Fixes

  • When leader change occurs in swarmkit the new leader’s node address can change to 0.0.0.0. The ucp-metrics inventory.json file may adopt a 0.0.0.0 target address as a result, thus producing a situation wherein the MKE dashboard is unable to display metrics for the leader node. (ENGORC-3256)

  • Adding in-house certificates to MKE can result in a reconciler loop when the cert’s issuer CN is empty. (ENGORC-3255)

  • Any LDAP search that returns 0 members (a normal result) results in the aborting of the entire LDAP sync. (ENGORC-3237)

3.3.0

(2020-05-19)

Components

Component

Version

MKE

3.3.0

Kubernetes

1.17.4

Calico

3.12.0

Interlock

3.1.0

Interlock NGINX proxy

1.14.2

What’s new

  • In previous versions, MKE did not include support for Kubernetes ingress, and that prevented the use of services running inside of the cluster from servicing requests that originated externally. Now, a service running inside the cluster can be exposed to service requests that originate externally using Istio Ingress, which means that you can route incoming requests to a specific service using Hostname, URL, and Request Header.

  • You can now use the following Istio ingress mechanisms:

    • Gateway

    • Virtual Service

    • Destination Rule

    • Mixer (handler, instance, and rule)

    • For this release, only Istio Ingress for Kubernetes is available.

  • MKE provides GPU support for Kubernetes workloads.

  • MKE supports Kubernetes on Windows Server nodes.

  • Windows Server Node VXLAN overlay network data plane in Kubernetes

    • Windows Server and Linux are now normalized to both use VXLAN.

    • Overlay networking applied to ensure communication between cluster nodes.

    • No longer requires BGP control plane (when using VXLAN).

    • Upgrade is not supported for VXLAN data plane. Only IPIP is supported. In this case windows nodes will not be part of the kube cluster.

  • In this release of MKE we upgraded the Kubernetes version used by MKE from 1.14.8 (in MKE 3.2) to 1.17.4 (in MKE 3.3). For information about Kubernetes 1.17, including a comprehensive list of new features, see Kubernetes 1.17 Release Notes.

  • The Kubernetes API underwent significant change with v1.16. For detailed information, refer to the release announcement. To learn about deprecated APIs that were removed in v1.16, refer to deprecated APIs. Significantly, as a result of the deprecation and removal of the “containerized” flag in Kubernetes, neither kubelet nor kube-proxy can run as container. For detailed information, refer to https://github.com/kubernetes/kubernetes/issues/74148.

Bug fixes

  • MKE 3.2 and earlier provided a global flag for granting users and service accounts permissions to deploy kubernetes workloads to a specific node. In MKE 3.3.0 we provide the ability to give specific permissions to users and service accounts by setting the kubernetes cluster-admin role bindings for just that user or service account.

    This change does not affect swarm workloads.

  • The feature gates VolumeSnapshotDataSource, ExpandCSIVolumes, CSIMigration, and VolumeSubpathEnvExpansion are included in the experimental storage features and are now enabled by default.

Known issues

  • Kubernetes Ingress cannot be deployed on a cluster after MKE is upgraded from 3.2.6 to 3.3.0. A fresh install of 3.3.0 is not impacted. This issue is internally tracked as FIELD-2602.

    • To reproduce this issue, upgrade a MKE 3.2.6 cluster to 3.3.0. Next, log into MKE in a browser window, navigate to Admin Settings, and select the Ingress tab from the left navigation. In the Ingress tab, set the following configuration options:

      When you click the “Save” button, the system fails to respond.

    • You can mitigate the problem by manually editing the MKE Configuration file.

      1. Download the ucp-config.toml file from MKE by following MKE Configuration File.

      2. Find the [cluster_config.service_mesh] heading and set it to true.

      3. Upload the modified ucp-config.toml file as described in MKE Configuration File.

      4. Navigate to MKE → Admin Settings → Ingress and change the parameters and enable Ingress.

      5. Click “Save” to enable Istio for the cluster.

  • To allow Windows worker nodes to join the cluster, all images must be pre-pulled. Refer to Use Kubernetes on Windows Server nodes.

  • The logs for the ucp-worker-agent-win container may warn that a certificate cannot be written because the file cannot be found. These warnings can be ignored. The container will be rescheduled automatically, within 90s, to repair this condition.

  • The MKE web interface has inconsistencies and unexpected behavior when interacting with the system. To work around this, use the command line tools when necessary.

  • You cannot configure VXLAN MTU and port on Windows Server. There is currently no workaround.

  • After vSwitch creation happens on windows node, connection to metadata server will be lost. It is known MSFT issue.

  • When kubernetes deployment is scaled up so that number of pods on Windows node is increased (e.g., kubectl scale --replicas=30 deployment.apps/win-nano) it sometimes cause nodes to become “not ready”. Related upstream issue: https://github.com/kubernetes/kubernetes/issues/88153. Frequency of the issue occurrence depends on node flavor (less vCPUs - more frequent) and on the scaling step (bigger step - more frequent). Please use node flavors with bigger vCPU count (8 or more) and smaller pods scaling step to reduce probability of the issue.

  • You may see a ‘Failed to compute desired number of replicas based on listed metrics’ in the Istio logs. You may ignore this error.

  • When reducing the number of gateways using the MKE web interface, the change will not take effect. The workaround is to toggle Istio off and back on again to lower the number of gateway replicas. Increasing replicas behaves appropriately, no workaround is needed for increases. CRDs from pre-existing Istio installation will be overwritten. To avoid this error, don’t install in clusters where Istio is already installed.

  • Resource types (instance, rule, and handler) can’t be created through the UI. The workaround is to perform the operation via kubectl.

  • The Kubernetes cloud provider for AWS is not enabled for Windows Server nodes. If MKE is installed with command: mirantis/ucp install --cloud-provider aws, the AWS cloud provider will be enabled for Linux nodes only. This issue does not affect Windows or Linux nodes on Azure (mirantis/ucp install --cloud-provider azure).

  • On Azure, the MKE installer may fail on the last step. Wait a minute and check the MKE UI to verify all nodes are healthy; the installer will continue when the nodes are ready.