Skip to content

Upgrade details#

Before you upgrade to MKE 4.2.1, review the changes detailed herein, as several require action before or after the upgrade. The number of supported upgrade sources has been restricted, cgroups v2 is now required on all nodes, and clusters carrying the legacy ingress-nginx addon go through an automated retirement flow. Several flags, restore behaviors, and authentication defaults have also changed in ways that may affect existing workflows.

Supported upgrade sources#

  • From MKE 3: only MKE 3.9.x is supported as an upgrade source for MKE 4.2.1. Upgrade to MKE 3.9.x first if you are on an earlier MKE 3 release.
  • From MKE 4: only MKE 4.1.3 or later is supported as an upgrade source for MKE 4.2.1. Clusters on 4.1.0 or 4.1.2 must first upgrade to 4.1.3 or later. Upgrading from 4.1.0 or 4.1.2 directly to 4.2.1 is blocked by the upgrade compatibility check.

cgroups v2 is required#

k0s 1.35 requires cgroups v2 on the host. Installs and upgrades now validate this requirement, and hosts that are still on cgroups v1 are rejected. Make sure all nodes are configured for cgroups v2 before you upgrade to MKE 4.2.1.

mkectl backup requires an encryption flag#

Action required before upgrade

Update every script, runbook, and automated job that runs mkectl backup without an encryption flag. Otherwise, these backups fail after you upgrade to MKE 4.2.1.

mkectl backup now requires one of the following flags:

  • --passphrase <value>: Encrypts the backup archive with the specified passphrase.
  • --no-passphrase: Creates an unencrypted backup archive, which matches the default behavior of earlier releases.

Previously, mkectl backup ran without either flag and created an unencrypted archive. To keep that behavior, add --no-passphrase to your existing commands:

mkectl backup --no-passphrase

For more information, refer to Create a backup.

Removed flags and changed behavior#

  • The --force-ignore-etcd-v2-data flag has been removed. The etcd v2 data carried over from MKE 3 is now cleaned up, WAL and data, during migration, which also resolves a class of upgrade failures caused by newer etcd versions rejecting the v2 API.
  • The internal v2 flag has been removed.
  • The mkectl restore command now restores the cluster to a single node, and in-place restore is no longer supported. After the restore completes, add the remaining control plane nodes and Linux worker nodes with mkectl node add, and add the Windows worker nodes with mkectl apply. For more information, refer to Restore.
  • mkectl apply no longer adds or removes control plane nodes on an existing cluster. If the control plane hosts in spec.hosts do not match the control plane nodes of the cluster, the command fails at the Check Managers Scaling Requested phase, before it makes any changes to the cluster. The command also fails for a replacement control plane node that reuses the host name or --hostname-override value of a previous node. To add or remove control plane nodes, use mkectl node add and mkectl node remove with --scope controller+worker. To replace a control plane node, remove the existing node with mkectl node remove, and then add the new node with mkectl node add.
  • mkectl apply now fails on an existing cluster if k0s is installed on a control plane node but is not running on any of the control plane nodes. To resolve the issue, start k0s on the existing control plane nodes or, if you have abandoned the installation, run mkectl reset.
  • mkectl backup now fails unless --passphrase or --no-passphrase is passed. The same requirement applies to mkectl node add and mkectl node remove for control plane nodes. Any automation that runs these commands must be updated before you upgrade mkectl.
  • Names in spec.kubelet.tlsCipherSuites are now validated at apply time. mkectl apply and the MKE operator reject unknown names, which previously passed mkectl apply and left the node in a NotReady state. Legacy CHACHA20 names that lack the _SHA256 suffix are also rejected, and therefore must be renamed before the upgrade.
  • The k0rdent UI has been removed. It is no longer deployed, the upgrade removes it from clusters on which it is present, and no setting is available to re-enable it.

Tigera Operator version managed by MKE 4#

Starting with MKE 4.2.1, the Calico CNI values.yaml must not set tigeraOperator.version. Child clusters follow this same rule through MkeChildConfig. For detail on the rule, refer to Tigera Operator version selection.

  • Upgrading from MKE 4.2.0 or earlier requires no action.

    If the configuration stored in the cluster pins an operator version, the upgrade removes the pin. A configuration that pins nothing is left unpinned, as the upgrade never adds the key.

  • Upgrading from MKE 4.2.1 or later with a pin present fails. This state is unsupported, and the upgrade reports but does not repair it:

    Error: failed to upgrade MKE to version <target-version>: failed to patch MKE config for target version <target-version>: unable to run migrations: phase 'Remove pinned tigera-operator version from Calico CNI config' failed: the Calico CNI config carries a pinned tigera-operator version that MKE <source-version> should not have: tigeraOperator.version must not be set (found "<pinned>"): remove the line -- MKE derives the tigera-operator version from the chart it ships (currently <shipped-version>)

Where:

- `<source-version>` is the version the cluster is upgrading from.
- `<target-version>` is the version the cluster is upgrading to.
- `<shipped-version>` is the Tigera Operator version the target release
  ships.

  • The upgrade does not modify your local configuration file.

    MKE 4.2.0 wrote the pin into its own generated Calico values, and as a result your local mke4.yaml can carry it even if it was never set deliberately. If your mke4.yaml pins an operator version, the upgrade succeeds and your next mkectl apply command run then fails, using the same file that worked before:

    pre-flight checks failed (1):
      ✗ tigera-operator version:
          invalid Calico CNI configuration: tigeraOperator.version must not be set (found "v1.37.1"): remove the line -- MKE derives the tigera-operator version from the chart it ships (currently v1.40.15)

To resolve this, delete the `version` line from the local file, or refresh
your local copy with the `mkectl config get` command.

NodeLocalDNS with a persisted configuration and a custom service CIDR#

Alert

Action may be required prior to upgrade.

Starting with MKE 4.2.1, MKE derives the kube-dns ClusterIP from spec.network.serviceCIDR rather than using a fixed default of 10.96.0.10. On upgrade, the node-local-dns DaemonSet is reconfigured to serve the derived address.

Clusters that set spec.nodeLocalDNS.nodeLocalConfigPersist: true are affected when the cluster also uses a non-default service CIDR. With nodeLocalConfigPersist enabled, MKE does not modify the existing node-local-dns ConfigMap, as the setting exists so that the Corefile can be managed outside of MKE. The persisted Corefile therefore continues to bind 10.96.0.10, which is no longer among the addresses the node cache owns, and the node cache Pods enter a crash loop reporting bind: cannot assign requested address. The MKE operator logs an error that describes the divergence and the remediation.

Clusters that use the default service CIDR, 10.96.0.0/12, are unaffected, as the derived address is the same 10.96.0.10.

This applies to MKE 4 to MKE 4 upgrades, and to MKE 3 to MKE 4 upgrades in which node_local_dns_config_persist = true carried the MKE 3 ConfigMap over.

To remediate, perform either of the following:

  • After the upgrade, edit the node-local-dns ConfigMap in the kube-system namespace and change every bind directive to bind 169.254.0.10 <kube-dns-ip>, where <kube-dns-ip> is the ClusterIP of the kube-dns Service.

  • Before the upgrade, temporarily set spec.nodeLocalDNS.nodeLocalConfigPersist to false so that MKE regenerates the Corefile. After the upgrade, re-enable the parameter and reapply any custom Corefile changes.

LDAP group search user matchers saved in reverse order#

Alert

Action may be required following upgrade.

Earlier versions of the MKE Dashboard saved the user attribute of an LDAP group search user matcher as groupAttr, and the group attribute as userAttr. MKE 4.2.1 corrects the form, however it cannot correct values that were saved previously. As the stored attributes remain reversed, group search can silently match no users.

Configurations written only through mke4.yaml are unaffected. If, however, the local mke4.yaml file was refreshed with mkectl config get after LDAP settings were saved in the MKE Dashboard, the file carries the reversed values as well.

If LDAP group search was configured through the MKE Dashboard, perform the following steps after the upgrade:

  1. Navigate to Admin Settings > Authentication > LDAP.
  2. Swap any reversed user matcher values.
  3. Click Save.
  4. If your local mke4.yaml was refreshed after the LDAP settings were saved in the MKE Dashboard, correct the user matchers in that file as well.

Inline authentication credentials moved to a Secret#

Alert

Action may be required prior to upgrading child clusters.

Starting with MKE 4.2.1, the inline ldap.bindPW, oidc.clientID, and oidc.clientSecret parameters are deprecated in MkeConfig and rejected in MkeChildConfig.

  • In MkeConfig, inline values generate a deprecation warning, and mkectl apply moves them into the authentication credentials Secret.
  • In MkeChildConfig, inline values are rejected once spec.version is set to 4.2.1. Before you upgrade a child cluster, move the values to a Secret and reference that Secret through spec.authentication.credentialsSecretName.

Shorter default login session lifetimes#

Starting with MKE 4.2.1, Dex applies the following token lifetimes when spec.authentication.expiry is not set:

  • ID tokens expire after 1 hour.
  • Refresh tokens expire after 4 hours of inactivity, or after 12 hours regardless of activity.

MKE 4.2.0 used the Dex defaults, which issue ID tokens that are valid for 24 hours and refresh tokens that never expire. Users therefore log in more often following the upgrade. To retain longer token lifetimes, set spec.authentication.expiry.

VRRP router ID and password change for control plane load balancing#

If spec.network.controlPlaneLoadBalancing.virtualRouterID is not set, the upgrade changes the VRRP router ID of the cluster from 51 to the last byte of the primary virtual IP address, and also changes the VRRP password. As the controllers restart one at a time during the upgrade, the API virtual IP address can move or become briefly unavailable.

Child clusters remain on their pinned MKE version#

Starting with MKE 4.2.1, upgrading the management cluster no longer rolls child clusters that are pinned to an earlier MKE version. Image tags, bootstrap scripts, the k0s binary, and k0s extension values are all gated by the MKE version of the child cluster. The Sveltos agent Pods on child clusters are, however, still replaced.

MkeChildConfig API changes#

The following changes have been made to the MkeChildConfig API:

  • The airgap.bastion and infrastructure.cloudName parameters have been removed.
  • The infrastructure.region parameter is deprecated in favor of infrastructure.configuration.region.
  • The image registry URL and the vSphere controlPlaneEndpointIP can no longer be changed after the child cluster is created.

MKE 3 to MKE 4 configuration migration#

The following changes apply to MKE 3 to MKE 4 upgrades:

  • Team and organization LDAP sync settings and noSimplePagination are migrated. Multiple LDAP user search filters are merged into a single filter, so that no users are dropped.
  • Directory-wide group-based user searches are not migrated, and the upgrade prints a warning when it encounters one. Following the upgrade, set spec.authentication.ldap.groupSearch manually, and then verify the migrated team and organization group assignments.
  • pre_logon_message is migrated to spec.authentication.preLogonMessage.
  • MKE 3 SAML and OIDC proxy settings are not migrated. Set spec.authentication.proxy following the upgrade.
  • The MKE 3 backup schedule is migrated. The cron expression is rewritten in MKE 4 form, backups_limit becomes retain, and include_logs carries over. If MKE 3 backups were encrypted, mkectl upgrade creates and labels the mke-backup-passphrase Secret from the MKE 3 passphrase. If you decline its creation, create the Secret before the first scheduled backup runs. The upgrade fails early if the MKE 3 schedule runs more often than every 5 minutes or uses @every.
  • MKE 3 clusters that run the Calico eBPF data plane continue to be rejected by the migration.
  • mkectl upgrade --output no longer writes MKE 3 secrets, namely the backup passphrase, SCIM token, SAML proxy password, and OIDC client secret, into mke3.toml or featureCompatibilityConfig.toml. The saved files are readable only by their owner.

Offline bundle and registry requirements for airgapped upgrades#

  • Upload the full MKE 4.2.1 offline bundle to your registry, as with earlier releases. The bundle carries every new image and chart, including the rbac-manager 1.21.5 chart, which MKE now also installs on child clusters.
  • The etcd 3.5.25 etcdctl and etcdutl binaries, which are used to clean up old etcd v2 data during upgrades from MKE 4.1.x, are no longer embedded in mkectl. They are now pulled from your registry as the etcdctl-v2 and etcdutl-v2 artifacts, and placed on the controller nodes by oci-downloader.
  • Authenticated registries must be served over HTTPS. mkectl never sends credentials to a plain http:// registry, and MKE 4.2.1 does not yet report a clear error in that case.
  • As the k0rdent UI has been removed, airgapped clusters no longer attempt to send k0rdent UI error reports to sentry.io.