Known issues#

The MKE 4 known issues with available workarounds are described herein.

Installation and node management#

Pod logs do not display when MKE 4 is unininstalled and then reinstalled on same nodes#

If you uninstall MKE 4 and later try to reinstall it on the same nodes, the installation will succeed but the pods that run on manager nodes from the previous installation will not display logs and will present the following CA error:

tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes-ca"

Workaround:

Following the uninstallation of MKE 4, reboot the manager nodes before you reinstall the software.
Run the following command on each manager node:
```
rm -rf /var/lib/kubelet/
```

Fresh installation interrupted when the admin kubeconfig is not yet written#

The mkectl apply command for a fresh installation is interrupted during the "Run k0sctl apply" phase, at the point where a k0s controller is up but the admin kubeconfig has yet to be written to ~/.mke/mke.kubeconf. Re-running mkectl apply then fails at the "Collect cluster facts" phase with the following message:

Error: apply failed: phase 'Collect cluster facts' failed: unable to initialize kube clients:
... stat /home/<user>/.mke/mke.kubeconf: no such file or directory

Workaround:

Important

The workaround detailed herein applies only to an interrupted fresh MKE 4.2.0 installation, as there is no running cluster or jeopardized data.

Run the mkectl reset --force command, to tear down the interrupted installation.
Run the mkectl apply -f mke4.yaml command, to re-run the fresh install.

Windows worker join hangs when mkectl is run over a high-latency link#

On clusters that have Windows worker nodes, the mkectl apply command can stall indefinitely during the Windows worker join when it is run from a host with high network latency to those nodes. The stall occurs as MKE 4 writes the k0s join token and a temporary kubeconfig to each Windows node. There is no phase timeout, so the command blocks until it is interrupted.

The issue occurs only when both of the following are true:

The cluster has one or more Windows worker nodes.
The round-trip latency between the host running mkectl and the Windows nodes is approximately 200 ms or greater.

Clusters with Linux worker nodes only are not affected, and the issue does not occur at LAN latencies.

Workaround:

Run the mkectl apply command from a host that has LAN-speed latency to the Windows nodes, such as a bastion host or an instance in the same region and network as the cluster.

Initial installation fails with unverified OIDC or proxy settings#

During the initial MKE 4 installation, the mkectl apply command fails if the spec.authentication.oidc or spec.authentication.proxy section of the mke4.yaml configuration file contains incorrect settings. The authentication-dex pods do not reach the Ready state, and the installation does not complete, preventing intial cluster provisioning from succeeding.

Workaround:

In the mke4.yaml configuration file, set enabled: false in the spec.authentication.oidc and spec.authentication.proxy sections. Leave the remaining settings in each section unchanged.
Run mkectl apply to install MKE 4.
After the installation completes, set enabled: true in the spec.authentication.oidc and spec.authentication.proxy sections.
Run mkectl apply again to enable OIDC and the authentication proxy.

Re-running an interrupted `mkectl apply` or MKE 3 upgrade makes Secrets unreadable#

If mkectl apply exits before it reports that MKE 4 installed successfully, for example because of an error or Ctrl-C, k0s can already be running on the nodes. Running mkectl apply again in this state replaces the Secret encryption key and the MKE client CA on the manager nodes. Although the second mkectl apply reports success, the API server cannot decrypt Secrets written before the second run once it restarts.

An MKE 3 to MKE 4 upgrade that stops without rolling back, for example because the mkectl upgrade process is killed, leaves the cluster in the same state. Running mkectl apply against that cluster replaces the encryption key migrated from MKE 3, and the migrated Secrets become unreadable.

Workaround:

Return the cluster to a clean state before you retry an installation or upgrade that did not report success:

For a new installation, run mkectl reset --force -f mke4.yaml, then run mkectl apply again.
For an MKE 3 to MKE 4 upgrade that neither completed nor rolled back, restore the MKE 3 cluster from your pre-upgrade backup, then upgrade again. Do not run mkectl apply or mkectl upgrade against the cluster until you restore MKE 3. For details, refer to Recover from an interrupted upgrade without rollback.

Node add or remove conflicts with etcd defragmentation#

When spec.etcd.maintenanceService.defragEnabled is true, etcd defragmentation can run while mkectl node add or mkectl node remove with --scope controller+worker changes etcd membership. The mkectl tool does not prevent this conflict.

Workaround:

Configure the etcd maintenance service so that defragmentation does not run during the node operation:

In the mke4.yaml configuration file, set spec.etcd.maintenanceService.defragEnabled to false.
Apply the configuration:
```
mkectl apply -f mke4.yaml
```
On a manager node, verify that no defragmentation is in progress:
```
curl http://localhost:18088/cluster/status
```
Confirm that the output reports DefragInProgress as false.
Add or remove the node.
Set spec.etcd.maintenanceService.defragEnabled to true and run mkectl apply again.

Upgrade from MKE 3.x#

Upgrade may fail on clusters with two manager nodes#

MKE 3 upgrades to MKE 4 may fail on clusters that have only two manager nodes.

Info

Mirantis does not sanction upgrading MKE 3 clusters that have an even number of manager nodes. In general, having an even number of manager nodes is avoided in clustering systems due to quorum and availability factors.

Kube-proxy IPVS mode is not supported#

Kube-proxy IPVS mode has been deprecated and its removal from Kubernetes is in progress. MKE 3 clusters running kube-proxy in IPVS mode will need to be retired or migrated to another kube-proxy mode. Please contact Mirantis support.

Upgrade to MKE 4 fails if kubeconfig file is present in source MKE 3.x#

Upgrade to MKE 4 fails if the ~/.mke/mke.kubeconf file is present in the source MKE 3.x system.

Workaround:

Make a backup of the old ~/.mke/mke.kubeconf file and then delete it.

MKE 3 clusters that use Calico in BPF mode cannot be upgraded to MKE 4#

You cannot upgrade MKE 3 clusters that are running Calico in BPF mode to MKE 4.

When an MKE 3 cluster being upgraded to MKE 4 has an LDAP root certificate configured through the rootCerts parameter, the migration maps the PEM certificate content into the Dex file path spec.authentication.ldap.ca rather than the raw certificate data file path spec.authentication.ldap.caData. As a result, the LDAP connector fails to initialize, and the following error is presented on MKE 4 LDAP login page:

Authentication error

Error type: Bad Request
Error message: Connector failed to initialize

Workaround:

Move the certificate content from ca to caData and clear the old ca value, so the LDAP connector initializes and logins can succeed.

Copy the value from ca to caData:

export KUBECONFIG=~/.mke/mke.kubeconf
CA=$(kubectl get mkeconfig mke -n mke -o jsonpath='{.spec.authentication.ldap.ca}')
kubectl patch mkeconfig mke -n mke --type merge \
  -p "{\"spec\":{\"authentication\":{\"ldap\":{\"caData\":$(jq -Rs . <<<"$CA")}}}}"

Remove the old ca key:

export KUBECONFIG=~/.mke/mke.kubeconf
kubectl patch mkeconfig mke -n mke --type merge \
  -p '{"spec":{"authentication":{"ldap":{"ca":null}}}}'

Namespace-scoped grants migrated from MKE 3 do not inherit team membership#

When you upgrade from MKE 3 to MKE 4, a namespace-scoped Kubernetes grant held by a team or organization is migrated with its members captured at the moment of the upgrade. The migrated grant is not reconciled against team membership afterward: users added to the team after the upgrade do not receive that grant's access in the namespace. Members present at the time of the upgrade are unaffected, and cluster-wide grants are unaffected — they continue to track membership through the team's aggregated role.

In the Grants view, the migrated grant is indistinguishable from a grant created natively in MKE 4. Name, role, and role kind match; only the namespace differs. Nothing in the MKE Dashboard indicates that the grant will fail to extend to new members.

As an example, if organization acme has a team dev granted the admin role in the dev-ns namespace during the upgrade, a user added to dev after the upgrade is denied access to dev-ns, despite the grant appearing valid.

Workaround:

Re-create the affected grant in MKE 4:

In the MKE Dashboard, navigate to Access Control > Grants.
Create a new grant with the same team, role, and namespace as the migrated one.

A grant created in MKE 4 tracks team membership, and thus all current and future members receive access. Repeat this for any team whose membership changes after the upgrade and that held a namespace-scoped grant in MKE 3. The original migrated grant is harmless and additive, and you can delete it once you have re-created the grant.

Upgrade from MKE 3 re-enables a disabled SAML configuration#

Disabling SAML on MKE 3 does not remove the idpMetadataURL and spHost values, and neither config-toml nor PUT /enzi/v0/config/auth/saml can clear them. Because the MKE 3 to MKE 4 conversion enables SAML whenever idpMetadataURL is set, the mkectl upgrade command re-enables SAML in MKE 4. If the mkectl host cannot reach the retained metadata URL, the upgrade fails at the Install MKE 4 Components step and rolls back.

Workaround:

Warning

Apply this workaround only to clusters on which SAML is disabled but SAML settings are still configured. The command deletes the SAML configuration, so do not run it on a cluster that still uses SAML.

Before you run mkectl upgrade, run the following procedure on the MKE 3 cluster:

Delete the SAML configuration:

curl --insecure -X DELETE -H "Authorization: Bearer $AUTHTOKEN" https://$MKE_HOST/enzi/v0/config/auth/saml

Verify that the command returns 204.

Run mkectl check mke3 and verify that the generated mke4.yaml configuration file contains saml.enabled: false.

MKE 3.9.5 upgrade to MKE 4.2.0 fails at `[Migrate MKE 3 Nodes]`#

Upgrades from MKE 3.9.5 using mkectl 4.2.0 fail at the [Migrate MKE 3 Nodes] point. MKE 3.9.5 ships with etcd 3.5.32, which rewords the etcdutl output that mkectl 4.2.0 matches on, so a genuinely clean v2 store is not recognized as clean. The upgrade fails safe and rolls back to MKE 3.9.5 with no data loss.

Note

The described issue only affects MKE 3.9.5, as earlier MKE 3.9.x releases carry an older etcd.
As the issue is fixed in MKE 4.2.1, Mirantis recommends that you upgrade with mkectl 4.2.1 or later if possible.

Workaround:

If you must upgrade with mkectl 4.2.0, refer to Runbook: MKE 3.9.5 to MKE 4.2.0 upgrade, etcd v2store-clean workaround.

Upgrade migrates MKE 3 team and organization grants to nonexistent users#

When you upgrade from MKE 3 to MKE 4.2.1, the upgrade migrates each grant that MKE 3 assigns to an organization or team onto a user, rather than onto the corresponding Organization or Team:

MKE 3 grant subject	MKE 4 grant subject
Organization	User `<org>`
Team	User `<org>-<team>`

Because these users do not exist, the migrated grants apply to no one, and members of the affected organizations and teams lose the access those grants provided. The upgrade summary reports the Authorization step as completed successfully and gives no indication of the failure.

To identify affected grants, review the MKE 4 Dashboard:

Users named <org> or <org>-<team> hold the migrated grants.
The corresponding Organizations and Teams show 0 grants.

Workaround:

To restore access, run the recover.sh script, which reassigns each affected grant to the matching Organization or Team. The script calls only the MKE 4 accounts API and requires curl, jq, and an MKE 4 administrator ID token.

Download recover.sh to a host that can reach the MKE 4 cluster.

View recover.sh

#!/bin/sh
# Moves MKE3 UI team/org grants that an affected MKE3->MKE4 migration
# (BOP-7096) left on users named "<org>" or "<org>-<team>" back onto the
# matching organization or team, using only the MKE4 accounts API.
#
# Dry-run by default; pass --apply to change the cluster. Safe to re-run.
#
# Usage:
#   MKE4_ADDR=<host[:port]> TOKEN=<id_token> ./recover.sh [--apply]
#   API=<base url, e.g. https://host/api/auth/v1> TOKEN_FILE=<path> ./recover.sh [--apply]
#
# TOKEN / TOKEN_FILE: id_token of an MKE4 admin (e.g. from the MKE4 dashboard login).
#
# Env:
#   WORKERS      parallel moves (default: 8)
#   EXCLUDE      space-separated user names that are real users; left alone
#   INSECURE=1   skip TLS verification (curl -k)
#   VERBOSE=1    print every planned move
#
# Requires: curl, jq, xargs.
set -eu

# Worker mode: move one grant. Invoked by xargs as "$0 __move <add_url> <del_url> <desc>".
if [ "${1:-}" = "__move" ]; then
    add=$(curl $CURL_OPTS -o /dev/null -w '%{http_code}' -X PUT -H "Authorization: Bearer $TOKEN" "$2") || add=000
    case $add in
    2??) ;;
    *) echo "FAIL add ($add): $4" >&2; exit 0 ;;
    esac
    del=$(curl $CURL_OPTS -o /dev/null -w '%{http_code}' -X DELETE -H "Authorization: Bearer $TOKEN" "$3") || del=000
    case $del in
    2?? | 404) echo "ok" ;;
    *) echo "FAIL remove ($del): $4" >&2 ;;
    esac
    exit 0
fi

APPLY=0
[ "${1:-}" = "--apply" ] && APPLY=1

CURL_OPTS="-sS --fail-with-body"
[ "${INSECURE:-0}" = 1 ] && CURL_OPTS="$CURL_OPTS -k"
export CURL_OPTS

if [ -z "${API:-}" ]; then
    : "${MKE4_ADDR:?set MKE4_ADDR or API}"
    API="https://$MKE4_ADDR/api/auth/v1"
fi
[ -z "${TOKEN:-}" ] && [ -n "${TOKEN_FILE:-}" ] && TOKEN=$(cat "$TOKEN_FILE")
: "${TOKEN:?set TOKEN or TOKEN_FILE to an MKE4 admin id_token}"
export TOKEN

get() { curl $CURL_OPTS -H "Authorization: Bearer $TOKEN" "$API$1"; }

tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

start=$(date +%s)
get /accounts/orgs >"$tmp/orgs.json"
get /accounts/teams >"$tmp/teams.json"
get /accounts/grants >"$tmp/grants.json"

# One line per move: add_url <TAB> del_url <TAB> description.
# A user name matching several org/team names (both contain "-") is skipped.
jq -r -n --arg api "$API" --arg exclude "${EXCLUDE:-}" \
    --slurpfile orgs "$tmp/orgs.json" --slurpfile teams "$tmp/teams.json" --slurpfile grants "$tmp/grants.json" '
    def q($g): [ (if $g.namespace then "namespace=\($g.namespace|@uri)" else empty end),
                 (if $g.roleKind == "Role" then "roleNamespace=\($g.namespace|@uri)" else empty end) ];
    def url($path; $params): $api + $path + (if ($params|length) > 0 then "?" + ($params|join("&")) else "" end);

    ( [ $orgs[0].orgs[]?   | {key: .name, value: {org: .name}} ]
    + [ $teams[0].teams[]? | {key: "\(.organization)-\(.name)", value: {org: .organization, team: .name}} ]
    ) as $targets
    | ($targets | group_by(.key) | map({key: .[0].key, value: map(.value)}) | from_entries) as $index
    | ($exclude | split(" ") | map(select(. != ""))) as $excluded
    | $grants[0].grants[]?
    | select(.kind == "user" and $index[.name] != null and (.name | IN($excluded[]) | not))
    | . as $g | $index[.name] as $t
    | if ($t|length) > 1 then
        "AMBIGUOUS\tuser \($g.name) \($g.roleKind):\($g.roleName) ns=\($g.namespace // "-") matches " + ([$t[] | if .team then "team \(.org)/\(.team)" else "org \(.org)" end] | join(", "))
      else
        $t[0] as $t
        | (if $t.team then url("/accounts/grants/\($g.roleName|@uri)/team/\($t.team|@uri)"; q($g) + ["orgName=\($t.org|@uri)"])
           else url("/accounts/grants/\($g.roleName|@uri)/org/\($t.org|@uri)"; q($g)) end) as $add
        | url("/accounts/grants/\($g.roleName|@uri)/user/\($g.name|@uri)"; q($g)) as $del
        | "\($add)\t\($del)\tuser \($g.name) \($g.roleKind):\($g.roleName) ns=\($g.namespace // "-") -> " + (if $t.team then "team \($t.org)/\($t.team)" else "org \($t.org)" end)
      end
' >"$tmp/plan.tsv"

grep '^AMBIGUOUS' "$tmp/plan.tsv" | cut -f2 | sed 's/^/SKIPPED, ambiguous, resolve by hand: /' || true
grep -v '^AMBIGUOUS' "$tmp/plan.tsv" >"$tmp/moves.tsv" || true
total=$(wc -l <"$tmp/moves.tsv" | tr -d ' ')
echo "orgs: $(jq '.orgs | length' "$tmp/orgs.json"), teams: $(jq '.teams | length' "$tmp/teams.json"), grants: $(jq '.grants | length' "$tmp/grants.json"); to move: $total grants (planned in $(($(date +%s) - start))s)"
[ "${VERBOSE:-0}" = 1 ] && cut -f3 "$tmp/moves.tsv" | sed 's/^/  /'

if [ "$APPLY" != 1 ]; then
    echo "dry-run: re-run with --apply to make these changes"
    exit 0
fi

start=$(date +%s)
# Tabs separate fields; xargs -d is GNU-only, so turn each line into three quoted args.
moved=$(awk -F '\t' -v q="'" '{ for (i = 1; i <= 3; i++) gsub(q, q "\\" q q, $i); print q $1 q " " q $2 q " " q $3 q }' "$tmp/moves.tsv" |
    xargs -n 3 -P "${WORKERS:-8}" "$0" __move 2>"$tmp/errors" | grep -c '^ok$' || true)
took=$(($(date +%s) - start))
failed=$(wc -l <"$tmp/errors" | tr -d ' ')
cat "$tmp/errors" >&2
echo "moved $moved of $total grants, $failed failed, in ${took}s"
[ "$failed" = 0 ]

Set the connection details for the MKE 4 cluster:
```
export MKE4_ADDR=<mke4-address>
export TOKEN=<admin-id-token>
```
To read the token from a file instead, set TOKEN_FILE=<path-to-token> in place of TOKEN. If the cluster uses a self-signed certificate, also set INSECURE=1.

Verify that the token grants access to the accounts API:

curl -s ${INSECURE:+-k} -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer ${TOKEN:-$(cat "$TOKEN_FILE")}" \
  https://$MKE4_ADDR/api/auth/v1/accounts/orgs

The command returns 200. Any other status code indicates that the address or token is incorrect.

Run the script in dry-run mode, which reports the planned moves without changing any grants:
```
chmod +x recover.sh
VERBOSE=1 ./recover.sh
```
Review the dry-run output:
- For each SKIPPED, ambiguous line, note the grant. The script does not move these grants, because organization and team names that contain - can match more than one Team.
- If the name of a real user matches an organization name or <org>-<team>, exclude that user and run the dry run again:
```
export EXCLUDE="<user-1> <user-2>"
VERBOSE=1 ./recover.sh
```
Move the grants:
```
./recover.sh --apply
```
The script moves up to 8 grants in parallel. To change this number, set WORKERS. When the run finishes, the script reports moved <N> of <N> grants, 0 failed. If any moves fail, run the script again. Already moved grants are not affected.
In the MKE 4 Dashboard, re-create each grant that the dry run reported as SKIPPED, ambiguous on the correct Team.
Verify the result:
- A dry run of ./recover.sh reports to move: 0 grants.
- The MKE 4 Dashboard shows the grants on the Teams and Organizations.
- The <org> and <org>-<team> users no longer appear in the MKE 4 Dashboard.

Upgrade between MKE 4 versions#

mkectl 4.1.5 backup command fails on clusters running versions earlier than MKE 4.1.5#

For mkectl 4.1.5 only, you cannot invoke the mkectl upgrade command with the --backup-path option to make use of a manually created pre-upgrade backup.

Workaround:

Users must rely on the automatic backup that is created during the upgrade procedure.

MKE 4.2.1 no longer deploys the k0rdent UI#

Starting with MKE 4.2.1, MKE 4 does not deploy the k0rdent UI. New clusters do not include the component, and upgrading from an earlier MKE 4 version removes the following resources from the k0rdent namespace:

The kcm-k0rdent-ui Deployment, ServiceAccount, Service, and ClusterRoleBinding
The kcm-nextauth Secret

Workaround:

Install the k0rdent UI through the MKE 4 spec.services mechanism, the same recommended mechanism that is used to install any additional service on an MKE 4 cluster. Unlike a standalone Helm release, this keeps the install under the mkectl apply command: editing or removing the services entry and re-applying updates or removes the release.

Warning

With basic authentication enabled, the Helm chart binds the UI ServiceAccount to the kcm-k0rdent-enterprise-global-admin-role ClusterRole. Every action you take in the UI runs with those permissions, which include cluster-wide access to Secrets and the ability to create and delete cluster deployments and credentials. If the UI does not require this level of access, set auth.basic.clusterRoleName to a more restrictive ClusterRole. In all cases, restrict network access to the k0rdent-ui Service.

Create a Secret that holds the UI password, in the k0rdent namespace:

kubectl create secret generic k0rdent-ui-basic-auth -n k0rdent \
  --from-literal=password='<your-password>'

Create a ServiceTemplate for the chart version, in the k0rdent namespace and reference it against the kcm-templates HelmRepository, which already exists on the management cluster for the k0rdent-enterprise chart registry (oci://registry.mirantis.com/k0rdent-enterprise/charts):
```
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ServiceTemplate
metadata:
  name: k0rdent-ui-1-2-1
  namespace: k0rdent
spec:
  helm:
    chartSpec:
      chart: k0rdent-ui
      version: 1.2.1
      interval: 1h
      sourceRef:
        kind: HelmRepository
        name: kcm-templates
```
For air-gapped environments, verify that your registry contains the k0rdent-ui image and chart from the Mirantis mirror bundle. Mirror any missing artifact from registry.mirantis.com:
- Image: registry.mirantis.com/k0rdent-enterprise/k0rdent-ui:1.2.1
- Chart: oci://registry.mirantis.com/k0rdent-enterprise/charts/k0rdent-ui, version 1.2.1
In preparation for step 4, set image.repository to the image location in your registry. If your registry requires authentication, also add an image.pullSecrets entry that references your registry pull secret:
```
image:
  repository: <your-registry>/k0rdent-enterprise/k0rdent-ui
  pullSecrets:
    - name: <secret>
```

Verify ServiceTemplate validation:

kubectl get servicetemplate k0rdent-ui-1-2-1 -n k0rdent -o jsonpath='{.status.valid}'

Add a services entry to your mke4.yaml configuration file, referencing the ServiceTemplate:

services:
  - name: k0rdent-ui
    namespace: k0rdent
    template: k0rdent-ui-1-2-1
    values: |
      image:
        repository: registry.mirantis.com/k0rdent-enterprise/k0rdent-ui
      auth:
        basic:
          enabled: true
          username: admin
          secretKeyRef:
            name: k0rdent-ui-basic-auth
            key: password
      nextAuth:
        generateSecret: true

The image.repository value shown is the public registry; for an air-gapped environment, substitute your mirrored image block from the previous step.

Apply the configuration:

mkectl apply -f mke4.yaml

Forward the k0rdent-ui Service port to your local machine. The Service is of type ClusterIP and listens on port 3000:
```
kubectl port-forward -n k0rdent svc/k0rdent-ui 3000:3000
```
Open http://localhost:3000 and log in as admin with the UI password.

`mkectl upgrade` may fail during a scheduled backup#

On clusters with spec.backup.schedule enabled, the mkectl upgrade command fails if its pre-upgrade backup starts at the same time that a scheduled backup is running on the same controller. Both operations share a node-local lock, and the upgrade does not wait for the lock to release.

When such a collision occurs, the upgrade aborts and rolls back automatically. The cluster remains healthy on its prior version, the backup schedule continues to run, and no data loss occurs.

Workaround:

Use one of the following methods:

Start the upgrade immediately after a scheduled backup completes, to maximize the time before the next scheduled backup.
Retry the upgrade if it fails.
Set spec.backup.schedule.enabled: false before you run the upgrade, and set it back to true after the upgrade completes.

MKE 4.1.x upgrade fails with calico-apiserver pods on worker nodes#

The upgrade from MKE 4.1.x to MKE 4.2.1 can fail at the Upgrade k0s step if both calico-apiserver pods are scheduled on the same worker node. Clusters with a single worker node are the most likely to encounter this issue.

During the upgrade, Calico migrates the calico-apiserver pods to a new namespace while MKE drains the worker node. The calico-apiserver pod disruption budget blocks eviction until the replacement pods are ready, which can cause the drain to exceed its five-minute timeout. The upgrade then fails with an error similar to the following and rolls back:

failed to run step [Upgrade k0s]: ... drain node: command failed: client exec: ssh session wait: Process exited with status 1

Workaround:

Before you run mkectl upgrade, move the calico-apiserver pods from the worker node to a manager node:

Identify which nodes the calico-apiserver pods are running on:
```
kubectl -n calico-apiserver get pods -o wide
```
If both pods run on the same worker node, evict them from that node, and then uncordon the node:
```
kubectl drain <worker-node> --pod-selector=k8s-app=calico-apiserver --ignore-daemonsets --timeout=10m
kubectl uncordon <worker-node>
```
The pod disruption budget ensures that Kubernetes evicts the pods one at a time.
Verify that both calico-apiserver pods are Ready on a manager node:
```
kubectl -n calico-apiserver get pods -o wide
```
Run mkectl upgrade.

Automatic pre-upgrade backup cannot be restored and is not deleted#

The mkectl upgrade command automatically creates a backup of the cluster, mke4preupgradesnapshot<timestamp>.tar, before it upgrades the cluster. The backup captures the pre-upgrade cluster state, but it identifies the new mkectl binary as its creator. As a result, no mkectl binary can restore the backup:

The earlier mkectl binary rejects the backup because a different version of mkectl created it.
The new mkectl binary bootstraps the node and then fails with errors such as .spec.tls: field not declared in schema.

This issue does not affect the automatic rollback of a failed upgrade, because rollback does not use mkectl restore.

In addition, mkectl upgrade does not delete the pre-upgrade backup after the upgrade completes. The backup is unencrypted and contains every Secret in the cluster.

Workaround:

Before you upgrade, use the mkectl binary that currently manages the cluster to back up the cluster, and store the backup together with that binary. If the upgrade is interrupted and the automatic rollback does not complete, use that binary to restore the backup. For more information, refer to Perform an MKE 4 to MKE 4+ Upgrade.
After the upgrade completes, delete the pre-upgrade backup from the directory in which you ran mkectl upgrade.

Upgrading a Calico eBPF cluster can fail and roll back#

On Calico eBPF clusters whose external address (spec.apiServer.externalAddress) is a DNS name, upgrading from 4.1.x or 4.2.0 to 4.2.1 can fail and roll back. Upgrades from 4.2.0 fail at the Upgrade CNI step; upgrades from 4.1.x fail at Upgrade k0s or Upgrade CNI, and in some test runs from 4.1.x the rollback also failed and left worker nodes out of the cluster. The Calico chart update removes the DNS setting that mkectl apply --bpf-dns-patch added to tigera-operator, so calico-node and tigera-operator cannot resolve the external address, and calico-node fails in install-cni with lookup <external-address>: i/o timeout. All failures seen so far were on AWS clusters installed with --bpf-dns-patch.

Workaround for management clusters:

Before you run mkectl upgrade, on every manager and worker node:

Find the external address IPs:

getent ahostsv4 <external-address> | awk '{print $1}' | sort -u

Add one line per IP to /etc/hosts:

echo "<LB-IP> <external-address>" | sudo tee -a /etc/hosts

Check that getent hosts returns one of those IPs, then run the upgrade.

Keep the entries after the upgrade. Later calico-node restarts need them, and so does any later mkectl apply that changes the Calico values, because that apply removes the DNS setting too. Update the entries if the load balancer IPs change. mkectl cannot reach child cluster nodes, so this workaround does not work for child clusters; an eBPF child cluster upgrade to 4.2.1 failed the same way in testing. Do not upgrade eBPF child clusters to 4.2.1.

Manager node drain fails after an upgrade from MKE 4.1.x#

After an upgrade from MKE 4.1.x, the MKE ingress proxy pods in the mke namespace match two PodDisruptionBudgets. As a result, eviction of these pods fails with the following error, and the kubectl drain command fails on manager nodes:

This pod has more than one PodDisruptionBudget

Ingress traffic is not affected.

Workaround:

Delete the envoy-mke- PodDisruptionBudget, ServiceAccount, and ConfigMap objects from the mke namespace:

kubectl -n mke get pdb,sa,cm -o name | grep '/envoy-mke-' | xargs -r kubectl -n mke delete

Child clusters#

Child clusters with AWS nodes can hang in unavailable state#

Due to an issue with the AWS cloud provider, MKE 4 child clusters can hang in an unavailable state for operations wherein the control plane nodes are scaled down (recreated or removed, for example).

Workaround method 1:

Restart the CAPI manager pod in the k0rdent namespace. On its recreation, the pod will reconfigure the connection to the child cluster and continue the scale-down process.

Workaround method 2:

Mnually change the configuration of Network Load Balancer (NLB) target groups; however, to do this, you must:

Locate the NLB of the child cluster in the AWS console.
Set the target group to the listener port 6443.
Enable Target deregistration management in the Attributes tab.

With this method, be aware that the target group can be reconciled by Cluster API Provider AWS (CAPA), which will result in the reversion of the configuration.

Child clusters default to dev registry when `registries` is omitted#

When a MkeChildConfig is applied without an explicit registries block, a child cluster is deployed using the development registry instead of the production registry.

Workaround:

Always set the registries block explicitly in MkeChildConfig.spec.

apiVersion: mke.mirantis.com/v1alpha1
kind: MkeChildConfig
metadata:
  name: my-mke-child
  namespace: k0rdent
spec:
  registries:
    chartRegistry:
      url: oci://registry.mirantis.com/mke
    imageRegistry:
      url: registry.mirantis.com/mke

Child clusters blocked as a result of Management cluster upgrade#

Child cluster deployments are blocked whenever the management cluster is upgraded to 4.2.0 from MKE 3.9.x or any MKE 4 version. As a result, some of the components that are needed to bootstrap and manage child clusters fail to install during the upgrade process.

Workaround:

Run the mkectl apply command to provision the missing components.

Child cluster component version unchanged after upgrade#

After a child cluster upgrade, a component of that cluster can remain at its pre-upgrade version. The child cluster reports as ready, all of its conditions are met, and the upgrade reports success. The component's version or image tag is the only indication of the problem.

The issue affects components that MKE 4 delivers to child clusters as k0s extensions. A child cluster upgrade replaces the control plane nodes. If a node is replaced while the cluster is applying one of these components, the component update is interrupted and does not resume. Because the issue depends on timing, it occurs in only a small number of upgrades.

To determine whether a component is affected, complete the following steps.

Step 1: Check whether the component is a k0s extension

Run the following command against the child cluster to list every component that k0s delivers as an extension, along with its running version and the version that MKE 4 expects:

kubectl get charts.helm.k0sproject.io -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,RELEASE:.status.releaseName,DEPLOYED IN:.status.namespace,RUNNING:.status.version,EXPECTED:.spec.version'

Example output:

NAMESPACE     NAME                       RELEASE     DEPLOYED IN      RUNNING     EXPECTED
kube-system   k0s-addon-chart-example-a  example-a   kube-system      0.0.10      0.0.10
kube-system   k0s-addon-chart-example-b  example-b   kube-system      2.62.0      2.64.0
kube-system   k0s-addon-chart-example-c  example-c   example-system   v3.31.700   v3.31.700

Review the output for the component that you are investigating:

If the component is not listed, it is not a k0s extension and this known issue does not apply.
If the RUNNING and EXPECTED versions differ, as in the example-b row, continue to the next step.

Step 2: Confirm the issue

A version mismatch alone does not confirm this issue. The update might still be in progress, or it might have failed for an unrelated reason.

Run the following command against the child cluster:

kubectl get charts.helm.k0sproject.io -A -o json | jq -r '
  .items[]
  | select((.status.error // "") != "" or .status.version != .spec.version)
  | "component: \(.metadata.namespace)/\(.metadata.name)
release:   \(.status.releaseName) in \(.status.namespace)
running:   \(.status.version)
expected:  \(.spec.version)
updated:   \(.status.updated)
error:     \(.status.error // "none")"'

Check the error field in the output for the following text:

another operation (install/upgrade/rollback) is in progress

Wait a few minutes, then run the command again and compare the results:
If the error and the updated timestamp are unchanged, the update is stalled. Proceed to Step 3.
If the command returns no output, the error text differs, or the updated timestamp has advanced, this known issue does not apply. No action is required for an update that is still progressing.

Step 3: Delete the interrupted release record

Using the release name and namespace from the step 2 output, list the release records for the component:

kubectl get secret -n <deployed-in-namespace> -l 'owner=helm,name=<release-name>' \
  -o custom-columns='NAME:.metadata.name,REVISION:.metadata.labels.version,STATUS:.metadata.labels.status' \
  --sort-by=.metadata.labels.version

Example output:

NAME                              REVISION   STATUS
sh.helm.release.v1.example-b.v1   1          superseded
sh.helm.release.v1.example-b.v2   2          deployed
sh.helm.release.v1.example-b.v3   3          failed
sh.helm.release.v1.example-b.v4   4          pending-rollback

Identify the record with the highest revision. Its status is pending-install, pending-upgrade, or pending-rollback, which marks the interrupted update that blocks all further attempts.

Warning

A healthy update in progress also shows a pending record. Delete the record only if step 2 returned the same error and the same updated timestamp on two checks several minutes apart. Never delete the record with the deployed status or the component entry listed in step 1, as doing so removes the component from the cluster.

Verify that an earlier record has the deployed status, then delete the pending record:

kubectl delete secret -n <deployed-in-namespace> <pending-record-name>

The update retries automatically within a few minutes. To confirm that the component reached its expected version, repeat step 2. The step 2 command returns no output when all components are up to date.

In MKE 4.2.1, AWS and vSphere child clusters derive the names of their machine templates from the child cluster name, truncated to 44 characters for worker nodes and to 48 characters for control plane nodes. As a result, two child clusters resolve to the same AWSMachineTemplate or VSphereMachineTemplate when all of the following conditions are met:

The child cluster names match in their first 43 characters.
At least one of the names is longer than 44 characters.
Both child clusters use the same infrastructure settings.

Both child clusters deploy without error. Deleting or upgrading either child cluster, or changing its infrastructure settings, removes the shared template, after which the other child cluster cannot create or replace machines. The issue does not affect child clusters that run MKE 4.2.0.

Workaround:

To prevent the issue, keep child cluster names to 44 characters or fewer, or make sure that no two child cluster names share their first 43 characters.

To identify and resolve existing shared templates:

List the machine templates that the worker nodes of each child cluster reference:

kubectl get machinedeployment -A \
  -o custom-columns=NAMESPACE:.metadata.namespace,CLUSTER:.spec.clusterName,TEMPLATE:.spec.template.spec.infrastructureRef.name

List the machine templates that the control plane of each child cluster references:

kubectl get k0scontrolplane -A \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TEMPLATE:.spec.machineTemplate.infrastructureRef.name

If the same TEMPLATE value appears for more than one child cluster, do not delete, upgrade, or change the infrastructure settings of those child clusters until a later MKE release resolves the issue. If you must modify an affected child cluster before then, contact Mirantis Support.

Dex fails to deploy on child clusters earlier than MKE 4.2.1#

On management clusters that run MKE 4.2.1 or later, the service controller fails to deploy the dex-http-server and authentication services to child clusters with spec.version set to v4.2.0 or earlier. The Helm charts in MKE 4.2.0 and earlier set the envFrom[].secretRef.namespace field, which the newer service controller rejects as undeclared.

To verify whether a child cluster is affected, run the following command on the management cluster:

kubectl -n k0rdent get clustersummary -o json | jq '.items[].status.helmReleaseSummaries'

The Helm release summaries for each affected child cluster include the following error:

envFrom[0].secretRef.namespace: field not declared in schema

Workaround:

Run the following steps against the management cluster:

Upgrade the child cluster to MKE 4.2.1:

kubectl -n k0rdent patch mkechildconfig <child-name> --type=merge \
  -p '{"spec":{"version":"v4.2.1"}}'

If the authentication service still fails with the same error after the upgrade, check the kcm controller logs for the following error:
```
failed to resolve services to apply: failed to determine upgrade paths:
failed to get upgrade paths for ServiceTemplate <template>: template <template> not found
```
The error occurs because the MKE 4.2.1 ServiceTemplateChain objects predate the MKE 4.2.0 ServiceTemplate objects. A chain that is created before its source template exists defines no upgrade path from that template, and kcm then rejects the entire service update.

In this event, delete the MKE 4.2.1 ServiceTemplateChain objects. The MKE 4 operator will recreate them with the MKE 4.2.0 templates as upgrade sources:
```
kubectl -n k0rdent delete servicetemplatechain \
  cert-manager-1-21-1 dex-http-server-chart-0-16-4 mke-aux-objects-0-4-4 \
  node-feature-discovery-0-19-0 mke-dashboard-chart-1-17-5
```

Within a few minutes, all services on the child cluster upgrade to their MKE 4.2.1 versions and the Helm feature in the ClusterSummary reports Provisioned. The Dex Pods remain in the ContainerCreating state until the mke-aux-objects service finishes upgrading, after which they start automatically. You do not need to change the Helm options or clean up the Helm release state.

MKE 4.1.x child clusters lose ingress after management upgrade#

The upgrade to MKE 4.2.1 includes k0rdent 1.4.1, which delivers a version of Sveltos that rejects the ingress chart that ships with MKE 4.1.x. MKE 4.2.1 protects the management cluster during its own upgrade. Child clusters that still run MKE 4.1.x, however, can lose ingress once the management cluster runs MKE 4.2.1, and they remain without ingress until you upgrade them.

Workaround:

Upgrade all MKE 4.1.x child clusters immediately after you upgrade the management cluster.

AWS child cluster names over 45 characters fail to become ready#

The MKE 4 operator accepts an AWS MkeChildConfig whose name exceeds 45 characters, but the resulting child cluster never reaches the Ready state.

Workaround:

Limit the names of AWS MkeChildConfig objects to 45 characters or fewer.

Note

AWS child clusters do not support air-gapped deployments.

Networking#

NodeLocalDNS on the Calico eBPF dataplane has no fallback to CoreDNS#

On a cluster that runs NodeLocalDNS with the Calico eBPF dataplane, pod DNS queries are directed to the node-local cache with no kube-proxy path behind it. If the node-local-dns Pod on a node stops serving, DNS resolution fails for all Pods on that node until the Pod recovers, typically within two minutes for a routine Pod replacement. A Pod that cannot recover leaves DNS unavailable on that node indefinitely. Pods on all other nodes are unaffected.

On the iptables dataplane, kube-proxy provides a fallback path to CoreDNS and this does not occur.

In addition, enabling NodeLocalDNS on an eBPF cluster interrupts DNS resolution on each node for up to approximately 45 seconds, while the node cache Pod starts. Disabling NodeLocalDNS does not interrupt DNS resolution.

NodeLocalDNS cannot be enabled or disabled during an upgrade on an eBPF cluster#

On a cluster that runs the Calico eBPF dataplane, changing spec.nodeLocalDNS.enabled in the same operation that changes the MKE version is rejected. Enabling or disabling the node-local cache has to be sequenced against the Calico eBPF NAT exclusion, and the upgrade flow does not perform that sequencing.

Workaround:

Complete the upgrade first, and then change spec.nodeLocalDNS.enabled in a separate run of the mkectl apply command.

Adding `bpfExcludeCIDRsFromNAT` on an eBPF cluster with NodeLocalDNS breaks networking#

On a Calico eBPF cluster that has NodeLocalDNS enabled, adding defaultFelixConfiguration.bpfExcludeCIDRsFromNAT to the Calico values.yaml file causes the mkectl apply command to fail at the Verify CNI step. During the apply operation, Felix loses its bpfEnabled setting. As a result, pod networking and cluster DNS stop working, and on a running cluster, ingress also stops working.

Workaround:

In the Calico values.yaml file, set bpfEnabled: true in the same defaultFelixConfiguration block as bpfExcludeCIDRsFromNAT:
```
defaultFelixConfiguration:
  enabled: true
  bpfEnabled: true
  bpfExcludeCIDRsFromNAT:
    - <your-cidr>
```
Note

Do not add the kube-dns /32 CIDR to the list. MKE 4 adds it automatically.
Run the mkectl apply command.

Verify the NAT exclusion list:

kubectl get felixconfiguration default -o jsonpath='{.spec.bpfExcludeCIDRsFromNAT}'

The output lists your CIDRs and the kube-dns /32 CIDR.

Port 5557 between manager nodes is not checked or opened#

The etcd-maintenance service uses port 5557/tcp for communication between manager nodes. The mkectl check ports command does not test this port, and the MKE 3 to MKE 4 upgrade does not open it on hosts that run a firewall. If the port is blocked, cluster-wide etcd defragmentation and backup replication fail.

Workaround:

Configure the firewall on each manager node so that it allows 5557/tcp traffic from the other manager nodes only.

Authentication#

Incorrect LDAP user matchers saved in the MKE 4.2.0 Dashboard after upgrade#

In MKE 4.2.0, the MKE 4 Dashboard swapped the two LDAP group search user matcher fields: it stored the user attribute value as groupAttr and the group attribute value as userAttr. MKE 4.2.1 fixes this issue, but values saved through the Dashboard in 4.2.0 remain incorrect after upgrading. Correct these values if necessary. Otherwise, LDAP group search cannot match groups, user tokens will not carry group membership, and permissions granted through LDAP groups have no effect.

Workaround:

Inspect the stored user matchers:
```
kubectl -n mke get mkeconfig mke \\
-o jsonpath='{.spec.authentication.ldap.groupSearch.userMatchers}'
```
Note

userAttr has a user entry attribute and groupAttr has a group membership attribute.
If necessary, correct a reversed pair:
1. log in to the MKE Dashboard as an administrator.
2. Navigate to Admin Settings > Authentication > LDAP, and in the Group Search section swap the two values in that user matchers row.
3. Click Save.
Repeat step 1 to verify the changes.

Certificates and TLS#

Expiring certificates can cause control plane failure#

Cluster certificates may expire a year after install or upgrade from MKE 3, rendering control plane components non-functional.

Workaround:

Restart the k0s controller service on the manager nodes to force certificate recreation. If you have a load balancer for manager nodes, you can manually force cert rotation without any visible downtime.

SSH into each manager node, one by one, and run the following commands:

Remove the etcd server and peer certificates to force renewal:

rm -f /var/lib/k0s/pki/etcd/server.crt
rm -f /var/lib/k0s/pki/etcd/server.key

Restart the k0s controller service:
```
systemctl restart k0scontroller
```

Dashboard kubeconfig contains outdated CA after custom certificate replacement#

On clusters that use custom TLS certificates, the MKE Dashboard reads the Kubernetes API server CA only at pod startup. If you replace the custom certificate and CA, kubeconfig files that you download from the Dashboard contain the previous CA in certificate-authority-data until the Dashboard pod restarts. Clients that use these kubeconfig files fail TLS verification against the API server.

Clusters that use the default MKE-managed certificates are not affected.

Workaround:

After you replace the custom certificate, restart the Dashboard so that it reads the new CA:

kubectl -n mke rollout restart deployment mke-dashboard
kubectl -n mke rollout status deployment mke-dashboard

Download the kubeconfig from the Dashboard again.
For any kubeconfig files downloaded before the restart, either download them again or replace the certificate-authority-data value with the new base64-encoded CA.

etcd fails to start if `spec.etcd.tlsCipherSuites` omits all AES-128-GCM suites#

If you set spec.etcd.tlsCipherSuites to a list that contains neither TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 nor TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, such as a list of only the AES-256 suites or only the ChaCha20 suites, etcd fails to start on every manager node. The mkectl apply command and the operator webhook both accept the configuration because neither validation path checks for an AES-128-GCM suite.

The Go HTTP/2 server that etcd uses requires at least one AES-128-GCM suite, and etcd exits with the following error:

Configure https server failed: http2: TLSConfig.CipherSuites is missing an HTTP/2-required AES_128_GCM_SHA256 cipher

Because mkectl apply restarts k0scontroller on all manager nodes in a single operation, all managers crash-loop at once and the Kubernetes API becomes unreachable. The apply operation times out and leaves behind a stale mke/cluster-operation lease that blocks subsequent apply operations.

Workaround:

To prevent the issue, either leave spec.etcd.tlsCipherSuites unset or include TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, or both in the list.

To recover an affected cluster:

In the mke4.yaml configuration file, add at least one AES-128-GCM suite to spec.etcd.tlsCipherSuites.
On each manager node, update spec.storage.etcd.extraArgs.cipher-suites in /etc/k0s/k0s.yaml to match the corrected list, and then restart k0s:
```
systemctl restart k0scontroller
```
After the Kubernetes API becomes responsive, delete the stale lease:
```
kubectl -n mke delete lease cluster-operation
```
Run mkectl apply.

Backup and restore#

Nodes must be rebalanced following restore operation#

Following cluster restore, all of the MKE 4 system components run on the single bootstrap node and are not automatically rebalanced when the remaining control plane nodes rejoin as a result of the mkectl apply command. As such, although all Pods report Running, critical management components remain on the bootstrap node.

Workaround:

To restore High Availability and to enhance node use, once all of the control plane nodes have rejoined the cluster and reported as Ready, force a rollout to reschedule the components across the nodes:

kubectl rollout restart deployment -n mke
kubectl rollout restart deployment -n k0rdent

Info

Rebalancing is not a factor for clusters that have a single controller node.

`mkectl restore` fails on a new host#

mkectl restore provides no option to set a new external address for the restored cluster. When you restore a backup onto a new host, which is the standard disaster-recovery scenario, and the archived externalAddress (DNS name or load balancer) does not point to that host, the restored node never reaches Ready status. The kube-proxy kubeconfig continues to reference the original address, the API server rejects its token, and CNI setup cannot reach the Kubernetes service cluster IP. The restore times out with the following error:

Error: failed to wait for bootstrap node to be ready: … node <name> is not ready: context deadline exceeded

Workaround:

Before you run the mkectl restore command, configure the archived externalAddress (DNS record or load balancer) so that it points to the restore host and routes port 443 to the restored controller. Joining the remaining nodes after the restore requires this connectivity.

Scheduled backups inactive after `mkectl restore`#

The mkectl restore command does not install or configure the etcd-maintenance service on the restored controller. As a result, the backup schedule defined in spec.backup.schedule remains inactive until mkectl apply completes successfully.

In disaster-recovery scenarios, mkectl apply fails if the external address of the cluster, be it a DNS name or a load balancer, does not yet point to the restored controller. The Wait for MKE addons to be ready phase requires the external address to respond to HTTPS requests on port 443.

Workaround:

After the restore completes, configure the cluster's external address so that it resolves to the restored controller. Alternatively, route port 443 to the MKE gateway locally on the restored controller.
Save the current cluster configuration and use it with the mkectl apply command. Do not apply the mke4.yaml file that mkectl restore writes once other control plane nodes have joined the cluster, as it lists only the restored node.
```
mkectl config get > mke4.yaml
mkectl apply -f mke4.yaml
```
Verify that the etcd-maintenance service is active:
```
systemctl is-active etcdmaintenance
```

`mkectl restore --force` does not skip the non-current-config prompt#

When you restore an archive whose bundled configuration differs from the current operator snapshot (OperatorLastKnownGood), mkectl restore is designed to require the --accept-noncurrent-config flag and to refuse non-interactive runs with restore not confirmed for an archive whose configuration is not the current operator snapshot. The --force flag does not suppress this confirmation prompt. Instead:

If stdin is closed, the command exits with failed to read input: EOF.
If stdin is open but is not a terminal, the command waits indefinitely for input.

No restore occurs in either case, so the safety behavior holds. However, scripted or unattended restores of non-current-config archives hang or fail instead of exiting cleanly.

Workaround:

In automation, review the provenance block, then pass --accept-noncurrent-config explicitly. To force a clean refusal instead, pipe n to stdin.

Backup fails when etcd database exceeds /tmp size#

The mkectl backup command stages the etcd snapshot in /tmp on the manager node before it streams the snapshot back. When /tmp is a tmpfs smaller than the etcd database, the backup fails, and MKE 4.2.0 reports only an SSH exit status rather than the missing space. The failure appears on clusters that backed up successfully before, once the etcd database has grown past the size of /tmp.

To work around the issue on MKE 4.2.0, either defragment etcd through the etcd maintenance service to shrink the database, or enlarge /tmp on the manager nodes.

Starting with MKE 4.2.1, you can set spec.backup.tempDir in the MKE 4 configuration, or pass --temp-dir, to stage the snapshot on a larger filesystem. For more information, refer to Backup: Temporary directory and disk space.

Cluster configuration and operations#

Post-install kubelet parameter modifications require a k0s restart#

Modifications made to the kubelet parameters in the mke4.yaml configuration file after the initial MKE 4 installation require a restart of k0s on every cluster node. To do this:

Wait for a short time, roughly 60 seconds after the application of the mkectl apply command, to give the pods time to enter their Running state.
Run the systemctl restart k0scontroller command on all manager nodes and the systemctl restart k0scontroller command on all worker nodes.

projectsveltos component can become stuck if it fails to deploy#

If the projectsveltos component fails to deploy when the mkectl apply command is run, for example because of temporary resource pressure on a node or a transient network hiccup, it can become stuck in a failed state from which it cannot recover on its own. When this happens, the command will not complete, even once the original cause has been cleared and mkectl apply is run again. Other cluster components are not affected.

Workaround:

Trigger a fresh reconcile of the projectsveltos release to clear the failed state. To do this, run the following against the cluster:

ts=$(date +%s)
kubectl --kubeconfig ~/.mke/mke.kubeconf -n k0rdent annotate helmrelease projectsveltos \
  reconcile.fluxcd.io/requestedAt="$ts" \
  reconcile.fluxcd.io/resetAt="$ts" \
  --overwrite

Once the release reconciles and reports as ready, mkectl apply will continue and successfully complete.

Enabling OPA Gatekeeper can fail when exempt namespaces do not exist#

When you enable the OPA Gatekeeper policy controller (spec.policyController.opaGatekeeper.enabled: true), the post-install hook job (gatekeeper-update-namespace-label) fails to label missing optional namespaces, leaving the gatekeeper addon in the Failed state.

Workaround:

Create the missing namespaces before you enable OPA Gatekeeper:

kubectl create namespace flux-system
kubectl create namespace windows-gmsa
kubectl create namespace blueprint-system

Reapply the mke4.yaml configuration file:
```
mkectl apply -f mke4.yaml
```

Windows gMSA admission webhook crash-loops on Linux manager nodes#

The Windows gMSA admission webhook in the windows-gmsa namespace runs on the Linux manager nodes. When a manager node exhausts its inotify instance limit, the webhook crash-loops with a too many open files error. Many Linux distributions set fs.inotify.max_user_instances to 128 by default.

Workaround:

Before you set spec.windows.gmsa.enabled to true, configure every Linux manager node so that it provides enough inotify resources:

Create /etc/sysctl.d/99-inotify.conf with the following content:

fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288

Load the new settings:
```
sudo sysctl --system
```

If the webhook is already crash-looping, raise the limits on every Linux manager node, and then restart the webhook pods:

kubectl delete pod -n windows-gmsa --all

Live API Documentation page is blank in air-gapped environments#

The Live API Documentation link in the MKE 4 Dashboard opens /api/auth/v1/docs, which loads Swagger UI from unpkg.com. Browsers that cannot reach the internet, as is typical in air-gapped environments, display a blank page. The MKE 4 API is not affected.

metrics-server does not run in high availability mode#

MKE 4 deploys metrics-server as a single instance. If the node that hosts metrics-server becomes unavailable, resource metrics are unavailable until metrics-server reschedules to another node.

mkectl CLI#

mkectl cannot run in parallel across multiple clusters due to shared $HOME paths#

mkectl writes all kubeconfig, logs, and local state to fixed paths under the invoking user's $HOME (for example, ~/.local/state/mke4-dsinfo/). Because these paths are not namespaced per cluster, concurrent mkectl commands targeting different clusters from the same host will overwrite each other's state.

As such, you cannot run operations against multiple clusters in parallel from a single host — even when the commands are independent (for example, mkectl apply against one cluster and mkectl upgrade against another from the same CI host). Installing multiple mkectl versions side-by-side does not resolve the issue, since the conflict is in the shared $HOME paths rather than in any per-binary state.

Workaround:

Set a unique $HOME for each cluster before invoking mkectl:

HOME=/tmp/cluster-a mkectl apply ...
HOME=/tmp/cluster-b mkectl upgrade ...

This isolates each cluster's kubeconfig, logs, and state to its own directory, allowing parallel installs and upgrades on the same host. Note that qa-mke already uses this pattern.

`mkectl apply` hangs on CRD schema validation errors#

When mke4.yaml sets a value that the MkeConfig CRD schema rejects but that mkectl does not pre-validate, mkectl apply does not fail immediately. Instead, it retries the change every 5 seconds until the 20-minute resource-request timeout expires, then fails with a generic context deadline exceeded error. At the default info log level, the command prints no output during the retries.

The change is not applied, and the cluster remains on its prior working configuration.

The following fields are validated only by the CRD schema:

spec.backup.schedule.encryption requires passphraseSecretRef.
spec.backup.historyRetain must be 1 or greater.
spec.backup.schedule.configSnapshotFallback accepts only None or LastKnownGood.

Workaround:

Run mkectl apply -l debug, or review the debug log from a previous run. The API server rejection message appears in the debug log on the first attempt, in the following format:

The MkeConfig "mke" is invalid: <field>: <reason>

Correct the field in mke4.yaml, then run mkectl apply again.

Known issues#

Installation and node management#

Pod logs do not display when MKE 4 is unininstalled and then reinstalled on same nodes#

Fresh installation interrupted when the admin kubeconfig is not yet written#

Windows worker join hangs when mkectl is run over a high-latency link#

Initial installation fails with unverified OIDC or proxy settings#

Re-running an interrupted mkectl apply or MKE 3 upgrade makes Secrets unreadable#

Node add or remove conflicts with etcd defragmentation#

Upgrade from MKE 3.x#

Upgrade may fail on clusters with two manager nodes#

Kube-proxy IPVS mode is not supported#

Upgrade to MKE 4 fails if kubeconfig file is present in source MKE 3.x#

MKE 3 clusters that use Calico in BPF mode cannot be upgraded to MKE 4#

Root certificate causes LDAP login failure after MKE 3 to MKE 4 upgrade#

Namespace-scoped grants migrated from MKE 3 do not inherit team membership#

Upgrade from MKE 3 re-enables a disabled SAML configuration#

MKE 3.9.5 upgrade to MKE 4.2.0 fails at [Migrate MKE 3 Nodes]#

Upgrade migrates MKE 3 team and organization grants to nonexistent users#

Upgrade between MKE 4 versions#

mkectl 4.1.5 backup command fails on clusters running versions earlier than MKE 4.1.5#

MKE 4.2.1 no longer deploys the k0rdent UI#

mkectl upgrade may fail during a scheduled backup#

MKE 4.1.x upgrade fails with calico-apiserver pods on worker nodes#

Automatic pre-upgrade backup cannot be restored and is not deleted#

Upgrading a Calico eBPF cluster can fail and roll back#

Manager node drain fails after an upgrade from MKE 4.1.x#

Child clusters#

Child clusters with AWS nodes can hang in unavailable state#

Child clusters default to dev registry when registries is omitted#

Child clusters blocked as a result of Management cluster upgrade#

Child cluster component version unchanged after upgrade#

Child clusters with similar long names can share a machine template#

Dex fails to deploy on child clusters earlier than MKE 4.2.1#

MKE 4.1.x child clusters lose ingress after management upgrade#

AWS child cluster names over 45 characters fail to become ready#

Networking#

NodeLocalDNS on the Calico eBPF dataplane has no fallback to CoreDNS#

NodeLocalDNS cannot be enabled or disabled during an upgrade on an eBPF cluster#

Adding bpfExcludeCIDRsFromNAT on an eBPF cluster with NodeLocalDNS breaks networking#

Port 5557 between manager nodes is not checked or opened#

Authentication#

Incorrect LDAP user matchers saved in the MKE 4.2.0 Dashboard after upgrade#

Certificates and TLS#

Expiring certificates can cause control plane failure#

Dashboard kubeconfig contains outdated CA after custom certificate replacement#

etcd fails to start if spec.etcd.tlsCipherSuites omits all AES-128-GCM suites#

Backup and restore#

Nodes must be rebalanced following restore operation#

mkectl restore fails on a new host#

Scheduled backups inactive after mkectl restore#

mkectl restore --force does not skip the non-current-config prompt#

Backup fails when etcd database exceeds /tmp size#

Cluster configuration and operations#

Post-install kubelet parameter modifications require a k0s restart#

projectsveltos component can become stuck if it fails to deploy#

Enabling OPA Gatekeeper can fail when exempt namespaces do not exist#

Windows gMSA admission webhook crash-loops on Linux manager nodes#

Live API Documentation page is blank in air-gapped environments#

metrics-server does not run in high availability mode#

mkectl CLI#

mkectl cannot run in parallel across multiple clusters due to shared $HOME paths#

mkectl apply hangs on CRD schema validation errors#

Re-running an interrupted `mkectl apply` or MKE 3 upgrade makes Secrets unreadable#

MKE 3.9.5 upgrade to MKE 4.2.0 fails at `[Migrate MKE 3 Nodes]`#

`mkectl upgrade` may fail during a scheduled backup#

Child clusters default to dev registry when `registries` is omitted#

Adding `bpfExcludeCIDRsFromNAT` on an eBPF cluster with NodeLocalDNS breaks networking#

etcd fails to start if `spec.etcd.tlsCipherSuites` omits all AES-128-GCM suites#

`mkectl restore` fails on a new host#

Scheduled backups inactive after `mkectl restore`#

`mkectl restore --force` does not skip the non-current-config prompt#

`mkectl apply` hangs on CRD schema validation errors#