Skip to content

Changes in behavior#

MKE 4.2.2 includes fixes that change behavior from MKE 4.2.1. The changes that require action before or after the upgrade are also listed in Upgrade details. If you upgrade from MKE 4.2.0 or earlier, also review the MKE 4.2.1 changes in Upgrade details.

etcd maintenance HTTP API removed#

Action required before upgrade

Update every script and automated job that calls localhost:18088. Otherwise, these calls fail after you upgrade to MKE 4.2.2.

The etcd maintenance service no longer serves an HTTP API on 127.0.0.1:18088, and its gRPC port, 5557, now requires a client certificate signed by the etcd CA.

Run the following commands on a controller node instead of calling the HTTP API. Use the full path to etcd-maintenance, because /usr/local/bin is not in the sudo path on RHEL.

HTTP API call Replacement command
GET /health sudo systemctl status etcdmaintenance
GET /cluster/status sudo /usr/local/bin/etcd-maintenance status
POST /cluster/defrag sudo /usr/local/bin/etcd-maintenance defrag --timeout-seconds <N> --pause-seconds <N>
POST /cluster/cleanup sudo /usr/local/bin/etcd-maintenance cleanup [--dry-run] [--show-keys] [--min-ttl-to-keep-seconds <N>] [--timeout 30m]

The spec.etcd.maintenanceService.httpPort parameter is deprecated and has no effect.

Flux resources removed from the built-in roles#

The built-in view, edit, and admin ClusterRoles no longer grant access to Flux resources, such as HelmRelease, HelmRepository, and Kustomization. The change applies to new installations, and to existing clusters at the next mkectl upgrade or mkectl apply command run.

Users and automation that read or manage Flux resources through the built-in roles receive Forbidden responses. Grant them an explicit role for the Flux API groups. Grant read access to HelmReleases in the k0rdent namespace only to cluster administrators.

New pre-flight and validation checks#

MKE 4.2.2 adds the following checks, which stop operations early that previously failed later:

  • A fresh installation, the mkectl node add command, and an MKE 3 upgrade fail at the pre-flight checks if firewalld is active or enabled on a Linux node. On an existing cluster, the mkectl apply command logs a warning instead. For more information, refer to Configure firewalld.
  • The mkectl upgrade command compares hosts.yaml with the MKE 3 Swarm before it changes anything. The command stops if a manager or worker is missing, a listed host is not in the Swarm, or a node is listed twice. The --force flag does not skip this check.
  • When the authentication proxy and OIDC are both enabled, the mkectl apply command checks from every controller that the OIDC issuer is reachable through the proxy. The check requires curl on the controllers. To skip the check, pass --skip-auth-proxy-check.
  • The mkectl node add command refuses to add controllers that would leave etcd with an even number of members. To add the controllers anyway, pass --force.
  • MKE 4 rejects a spec.etcd.tlsCipherSuites list that contains neither TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 nor TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256.
  • MKE 4 rejects MkeChildConfig names longer than 45 characters for child clusters.

Scheduled backups skip a missing mount#

Before a scheduled backup writes its archive, MKE 4 checks that the filesystem expected at spec.backup.schedule.path is mounted. If the filesystem is not mounted, MKE 4 writes no backup, records the MkeBackup as Skipped, and retries at the next scheduled run.

Declare the backup mount in /etc/fstab on every controller node, so that MKE 4 can detect a missing mount from the first backup.

mkectl upgrade --disable-rollback is supported#

The mkectl upgrade command now supports the --disable-rollback flag for MKE 3 and MKE 4 upgrades. When a step fails, mkectl collects a support bundle, reports the failed step, and exits without rolling back. The cluster remains partly upgraded. The pre-upgrade backup is still taken.

mkectl restore --bpf-dns-patch#

The mkectl restore command now takes the --bpf-dns-patch option, as do the mkectl apply and mkectl upgrade commands. Pass the option when you restore a Calico eBPF cluster on AWS. mkectl restore never infers the option from the backup. For restores on other infrastructure, refer to Known issues.

Cloud provider install flags and node initialization#

With the cloud provider enabled, the mkectl apply and mkectl node add commands now keep the cloud provider install flags. Each mkectl apply command run taints nodes that have no ProviderID with node.cloudprovider.kubernetes.io/uninitialized until the cloud controller manager initializes them.

If a node remains tainted, fix the cloud controller manager configuration. Do not remove the taint, because the next mkectl apply command run adds it again.

Child cluster nodes no longer contact the management cluster#

Child cluster nodes at MKE 4.2.2 download oci-downloader from the image registry instead of the management cluster API. The nodes no longer need to reach the management cluster API server during bootstrap, scaling, or upgrades. Child clusters at earlier versions keep the previous behavior.

In airgapped environments, the oci-downloader binaries artifact must be in the registry that the child cluster uses. For more information, refer to Upgrade details.

Other changes#

  • When the k0s configuration changes, the mkectl apply and mkectl upgrade commands restart the k0s controllers one at a time.
  • The MKE Dashboard /env.json file no longer contains certificateAuthorities. To retrieve the CA certificates, use the /ca and /cluster-ca endpoints instead.