Skip to content

Addressed issues#

Issues addressed in the MKE 4.2.1 release include:

Backup and restore#

  • You can now set where k0s stages backup and restore data. Use the --temp-dir flag with the mkectl backup, mkectl restore, and mkectl upgrade commands, or set spec.backup.tempDir. For backup, the directory needs free space at least equal to the size of the etcd database. For restore, it needs free space at least equal to the uncompressed size of the k0s archive.
  • Rolling back a failed upgrade now re-applies custom node annotations, in addition to node labels, on all nodes. Restoring a backup onto new nodes still does not preserve node labels or annotations that you set outside the MKE configuration.
  • The first mkectl apply command run after the mkectl restore command is run no longer reinstalls k0s on every controller.
  • Restore now preserves the cloud provider install flags, so the first mkectl apply command run after a restore no longer reinstalls k0s.
  • Controller restore and node add operations now succeed on clusters with custom TLS when you use an mke4.yaml configuration file generated by the mkectl config get command.
  • Restore and upgrade rollback no longer fail permanently when a container rootfs mount persists after k0s stop.
  • Restore no longer fails with kubernetes not ready: context deadline exceeded when the restored controller is slow to start the API server. Restore and rollback now wait up to 15 minutes, and the error message names the likely cause.
  • The mkectl restore command no longer adds its context to the kubeconfig file named by $KUBECONFIG or switches the current context of that file, which caused later kubectl commands to target the restored cluster. The mkectl apply, mkectl restore, and mkectl upgrade commands now leave untouched any kubeconfig files they did not create.
  • Fixed an issue wherein the restore operation could apply the restored configuration to a cluster other than the one it had just bootstrapped. This occurred when the external address in the backup archive still resolved to a different, running cluster. Restore now verifies that its kubeconfig targets the newly bootstrapped cluster and stops with an error if it does not.
  • MKE 4 system components no longer remain on the bootstrap controller after a restore. The mkectl apply command run that rejoins the other controllers now distributes them across the control plane.

Upgrade and migration#

  • MKE 4 now removes the legacy Calico namespace only after calico-apiserver is Ready in its new namespace. Otherwise, it leaves the legacy namespace in place and reports this in the upgrade summary.
  • Fixed an issue wherein the rendered Calico values contained a fixed tigeraOperator.version override, which prevented MKE from managing the Calico version. MKE 4.2.1 removes the override from the rendered values, and upgrades from earlier MKE 4 releases remove it from the stored configuration. For more information, refer to Upgrade details.
  • The MKE 3 to MKE 4 migration no longer times out and rolls back when an unfinished etcdv3 to KDD migration has left the MKE 3 Calico datastore locked. A pre-flight check now stops the upgrade before it starts and instructs you to run calicoctl datastore migrate unlock.
  • The MKE 3 to MKE 4 upgrade no longer fails during CRD removal on clusters with MetalLB or Multus enabled.
  • The MKE 3 LDAP migration now writes PEM root certificates to ldap.caData instead of the ldap.ca path field.
  • Teams and organizations whose membership was managed by LDAP in MKE 3 now continue to sync with LDAP after the MKE 3 to MKE 4 upgrade. Previously, membership stayed as it was on the day of the upgrade: users added to the LDAP group did not get the team permissions, and users removed from it retained them. The upgrade summary now reports the result of the LDAP sync migration.
  • The MKE 3 to MKE 4 upgrade now combines all MKE 3 LDAP user and group search filters into one Dex search. Previously, some of the filters were dropped, so only some LDAP users were migrated.
  • Users who were direct members of an MKE 3 organization now display as organization members in the MKE 4 Dashboard following the upgrade. Previously, the migration created no binding for them.
  • Cluster-wide user grants migrated from MKE 3 now display on the Grants page. Previously, they still took effect but were not shown.
  • Legacy CHACHA20 cipher names carried over from MKE 3 no longer cause mkectl configuration validation to block the upgrade. MKE 4 now renames them to their _SHA256 form.
  • The mkectl upgrade command now publishes the oci-downloader chunk ConfigMaps, so management clusters upgraded from 4.1.x can provision new child clusters.
  • Upgrading from MKE 4.1.x no longer fails with a k0rdent timeout caused by envoy-gateway crash-looping. MKE 4 no longer renders an envoy-gateway deploy block without a type, and k0rdent wait timeouts now name the components that are not ready.
  • A single transient SSH failure during upgrade rollback no longer orphans controllers on multi-manager clusters. Node connections now retry for up to 5 minutes during upgrade, rollback, and restore.

Child clusters#

  • The mkectl apply command no longer times out at Setup K0rdent on management clusters that have child clusters.
  • The etcd-maintenance defragmentation and cleanup operations now succeed on child clusters. Previously, these operations failed because the leader lease stored the OS hostname.
  • Rotating a user-supplied encryption-config or audit-webhook Secret for a child cluster now rolls the machines that use it.
  • Changing spec.registries.imageRegistry.url on an existing child cluster no longer leaves the control plane roll stuck. Previously, the change rewrote k0s system images to nonexistent paths. You can no longer change the image registry URL after you create the child cluster.

Networking and load balancing#

  • Control plane load balancing no longer uses the same VRRP router ID (51) and password on every cluster, which caused conflicts between clusters on the same L2 segment. Each cluster now derives its router ID from the last byte of its virtual IP, and mixes the virtual IP into its password as well. Previously, the password depended only on the MkeConfig resource namespace and name — the same on every cluster by default — and thus every cluster shared the same password regardless of its virtual IP. Alternatively, you can set virtualRouterID. For more information, refer to Control Plane Load Balancer (CPLB).
  • MKE 4 no longer reads the port in apiServer.externalAddress as the kube-apiserver port, which broke the Calico bootstrap endpoint.
  • The MkeConfig webhook now accepts a custom kubeletRootDir when the CNI values set installation.kubeletVolumePluginPath.
  • NodeLocalDNS now derives the kube-dns ClusterIP from spec.network.serviceCIDR. Previously, MKE 4 configured the node-local DNS cache with the default kube-dns address, 10.96.0.10, regardless of the configured service CIDR. On clusters with a non-default service CIDR, the node-local-dns DaemonSet reported a healthy status but did not intercept pod DNS traffic, and DNS caching was inactive with no error reported.
  • In airgapped installations, oci-downloader now resolves the registry name when DNS returns NXDOMAIN for the IPv6 (AAAA) record but a valid IPv4 record exists. Previously, the mkectl apply command stopped at Download k0s from registry. oci-downloader is updated to v0.3.4, and the mkectl upgrade command ships the new version to the cluster.

Security#

  • MKE 4 no longer deploys the k0rdent UI. The k0rdent UI used a fixed login (admin/k0rdent), and its service account was bound to the k0rdent global-admin ClusterRole. This change also stops the k0rdent UI error reports to sentry.io.
  • The mkectl apply -l debug command no longer writes kubelet client private keys to the log.
  • The mkectl upgrade --output command now redacts MKE 3 secrets, including the backup passphrase and SCIM token, and makes the output files readable only by their owner. Previously, it wrote these values in plain text to files that any local user could read.
  • At the debug log level, the mkectl apply command now masks the Dex admin password hash in the MKE operator chart values.
  • The accounts API server (dex-http-server) now sets HTTP timeouts, so slow clients can no longer hold connections open indefinitely.
  • Dex refresh tokens no longer accumulate during concurrent logins. MKE 4 now removes unreferenced tokens every hour.
  • Listing organizations no longer returns 504 Gateway Timeout on clusters with many organizations and teams.
  • Users with access to only some namespaces now see the grants for the namespaces where they have full access on the Grants page, instead of the error Server error - unable to view grants.
  • The accounts API no longer returns a 500 error for a team's LDAP user search and sync when the team has no LDAP search configured.
  • A new cluster with LDAP enabled in the initial mke4.yaml configuration file now deploys Dex and the dashboard. Previously, mkectl apply succeeded, but the operator failed with Secret "authentication-credentials" not found and neither Dex nor the dashboard was available.
  • MKE 4 now reports an error when waiting for a Dex user to be created fails.
  • MKE 4 now reports an error instead of panicking when ssoMetadataURL points to SAML metadata with no IdP descriptor, such as service provider metadata.
  • The dashboard LDAP form now saves the two user matcher columns to the correct fields, so LDAP group-based permissions apply. For clusters already affected, verify the groupSearch.userMatchers values against LDAP and correct them if necessary.

mkectl and operations#

  • The mkectl apply command now waits for newly enabled addons to reach the Deployed state before it returns. It also checks the MKE external endpoint and Pod readiness in the mke namespace.
  • Retrying the mkectl apply command after an interruption during k0s installation no longer fails at Collect cluster facts.
  • The mkectl apply command no longer exits with code 0 after catching a panic or leaves the cluster-operation lease held.
  • The mkectl apply command on an existing FIPS cluster no longer fails the pre-flight operator version check.
  • Re-running mkectl node add or mkectl node remove for a host already in the requested state no longer runs the full flow, and mkectl node add no longer runs installation steps against the helper controller.
  • The mkectl apply command no longer spends more than an hour in Apply SELinux labels on large clusters.
  • The mkectl init command no longer generates grafana and ingressController settings.
  • etcd v2store cleanup timeout messages now name the last node and include the etcdutl output.
  • With a custom kubeletRootDir, dcgm-exporter GPU metrics now include pod and container labels. MKE 4 now passes the kubelet root directory to the GPU Operator.

Windows#

  • The Secrets Store CSI Driver Windows DaemonSet is now schedulable. Previously, the kubelet host paths used the Linux root path.
  • Stale HNS state after k0s reset no longer leaves Windows pods stuck in ContainerCreating after a node rejoins.
  • Windows host preparation no longer accepts a readiness probe from a host that is still within its reboot window.
  • The mkectl support command now collects Windows container logs.
  • Pods on a joining or restoring Windows node no longer start before calico-node-windows resets its network and then fail with endpoint was not found. mkectl apply now keeps workload pods off Windows nodes until calico-node-windows is ready.

MKE 4 Dashboard#

  • The MKE 4 version now displays for users who are not cluster admins, instead of --not available--.
  • The About page now displays for LDAP cluster-admin users.
  • Lists no longer appear empty and without page controls when opened at a page number past the end of the list, for example after you select a namespace filter from a later page of another list.
  • The grants table columns now read Subject name and Subject kind. Previously, the table presented the subject's name and kind as if they belonged to the grant.
  • After the ingress CA changes, the kubeconfig downloaded from the dashboard now uses the new CA without a dashboard pod restart.
  • Previewing a team's LDAP user search no longer saves the search or changes the team's members. The dashboard now also validates the LDAP filter before a sync.
  • The Controllers bulk Actions menu is now consistent with the per-item delete action.