Skip to content

Known issues#

The MKE 4 known issues with available workarounds are described herein.

Post-install kubelet parameter modifications require a k0s restart#

Modifications made to the kubelet parameters in the mke4.yaml configuration file after the initial MKE 4 installation require a restart of k0s on every cluster node. To do this:

  1. Wait for a short time, roughly 60 seconds after the application of the mkectl apply command, to give the pods time to enter their Running state.

  2. Run the systemctl restart k0scontroller command on all manager nodes and the systemctl restart k0scontroller command on all worker nodes.

Upgrade may fail on clusters with two manager nodes#

MKE 3 upgrades to MKE 4 may fail on clusters that have only two manager nodes.

Info

Mirantis does not sanction upgrading MKE 3 clusters that have an even number of manager nodes. In general, having an even number of manager nodes is avoided in clustering systems due to quorum and availability factors.

Kube-proxy IPVS mode is not supported#

Kube-proxy IPVS mode has been deprecated and its removal from Kubernetes is in progress. MKE 3 clusters running kube-proxy in IPVS mode will need to be retired or migrated to another kube-proxy mode. Please contact Mirantis support.

Upgrade to MKE 4 fails if kubeconfig file is present in source MKE 3.x#

Upgrade to MKE 4 fails if the ~/.mke/mke.kubeconf file is present in the source MKE 3.x system.

Workaround:

Make a backup of the old ~/.mke/mke.kubeconf file and then delete it.

MKE 3 clusters that use Calico in BPF mode cannot be upgraded to MKE 4#

You cannot upgrade MKE 3 clusters that are running Calico in BPF mode to MKE 4.

reset command must be run with --force flag#

You must run the reset command with the --force flag, as without this flag the command will always return an error.

mkectl reset -f mke4.yaml

Example output:

time="2025-09-08T19:35:44-04:00" level=info msg="==> Running phase: Disconnect from hosts"
Error: reset requires --force

Addition of extra scopes with mkectl login causes CLI to authenticate twice#

When you create the kubeconfig with the mkectl login command and add extra scopes using the --oidc-extra-scopes flag, the MKE 4 CLI attempts to authenticate two times during the generation of the configuration and on each cluster interaction with the generated kubeconfig.

Workaround:

When adding extra scopes to the --oidc-extra-scopes flag, make sure to also add the offline_access scope. For example:

--oidc-extra-scopes=groups,offline_access`

mkectl config get command generates log lines that malform YAML output#

The mkectl config get command output contains log records at the beginning of the generated output that invalidate the resulting YAML configuration file.

Workaround:

Exclude unwanted logs by running the mkectl config get command with the higher log level.

mkectl config get -l fatal

Pod logs do not display when MKE 4 is unininstalled and then reinstalled on same nodes#

If you uninstall MKE 4 and later try to reinstall it on the same nodes, the installation will succeed but the pods that run on manager nodes from the previous installation will not display logs and will present the following CA error:

tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes-ca"

Workaround:

  1. Following the uninstallation of MKE 4, reboot the manager nodes before you reinstall the software.

  2. Run the following command on each manager node:

    rm -rf /var/lib/kubelet/

Child clusters with AWS nodes can hang in unavailable state#

Due to an issue with the AWS cloud provider, MKE 4 child clusters can hang in an unavailable state for operations wherein the control plane nodes are scaled down (recreated or removed, for example).

Workaround method 1:

Restart the CAPI manager pod in the k0rdent namespace. On its recreation, the pod will reconfigure the connection to the child cluster and continue the scale-down process.

Workaround method 2:

Mnually change the configuration of Network Load Balancer (NLB) target groups; however, to do this, you must:

  1. Locate the NLB of the child cluster in the AWS console.

  2. Set the target group to the listener port 6443.

  3. Enable Target deregistration management in the Attributes tab.

With this method, be aware that the target group can be reconciled by Cluster API Provider AWS (CAPA), which will result in the reversion of the configuration.

Use of --keep-ingress-nginx flag during upgrade requires two correlated flags#

To apply the --keep-ingress-nginx flag during an upgrade from a previous MKE 4 version to MKE 4.1.3 you must also apply both the --ingress-nginx-alt-http-node-port flag and the --ingress-nginx-alt-https-node-port flag, otherwise, the target cluster will not be fully functional.

Long upgrade times from MKE 4.1.2 to version 4.1.5#

The upgrade from MKE 4.1.2 to version 4.1.5 may appear to stall while waiting for the Management resource to become ready, as this step can take a significant amount of time.

Example output:

DBG checking Management Ready condition
DBG k0rdent installation is not Ready, components status:
DBG capi: true
DBG kcm: true
DBG cluster-api-provider-aws: true
DBG cluster-api-provider-gcp: true
DBG cluster-api-provider-k0sproject-k0smotron: true
DBG cluster-api-provider-vsphere: true
DBG cluster-api-provider-azure: InfrastructureProvider is not yet ready: condition Ready is in status False
DBG mke-operator: true
DBG projectsveltos: true
DBG cluster-api-provider-docker: true
DBG cluster-api-provider-infoblox: true
DBG cluster-api-provider-ipam: true
DBG cluster-api-provider-openstack: true

No action is required. The upgrade will automatically proceed after 25–35 minutes.

Workaround:

To avoid the long wait, you can use kubectl to disable CAPI providers prior to running the upgrade.

Warning

Do not disable CAPI providers if you have child clusters as this will result in the loss of those child clusters.

kubectl patch management kcm \
  --type='json' \
  -p="$(kubectl get management kcm -o json | \
    jq -c '[{"op":"replace","path":"/spec/providers","value":(.spec.providers | map(select(.name == "projectsveltos" or .name == "mke-operator")))}]')"

Use of Calico CNI with BPF dataplane requires customer intervention#

Due to a known Calico CNI issue, MKE 4.1.5 customers using Calico CNI with BPF dataplane must disable bpfConnectTimeLoadBalancing in the defaultfelixConfiguration configuration of the Calico CNI configuration file:

   defaultFelixConfiguration:
     enabled: true
     bpfConnectTimeLoadBalancing: Disabled
   ....

For fresh MKE 4 installations, this must be done at the time the cluster is created. To upgrade existing MKE 4.1.2 clusters with BPF dataplane to version 4.1.5, this must be done prior to the upgrade.

mkectl 4.1.5 backup command fails on clusters running versions earlier than MKE 4.1.5#

For mkectl 4.1.5 only, you cannot invoke the mkectl upgrade command with the --backup-path option to make use of a manually created pre-upgrade backup.

Workaround:

Users must rely on the automatic backup that is created during the upgrade procedure.

mkectl cannot run in parallel across multiple clusters due to shared $HOME paths#

mkectl writes all kubeconfig, logs, and local state to fixed paths under the invoking user's $HOME (for example, ~/.local/state/mke4-dsinfo/). Because these paths are not namespaced per cluster, concurrent mkectl commands targeting different clusters from the same host will overwrite each other's state.

As such, you cannot run operations against multiple clusters in parallel from a single host — even when the commands are independent (for example, mkectl apply against one cluster and mkectl upgrade against another from the same CI host). Installing multiple mkectl versions side-by-side does not resolve the issue, since the conflict is in the shared $HOME paths rather than in any per-binary state.

Workaround:

Set a unique $HOME for each cluster before invoking mkectl:

HOME=/tmp/cluster-a mkectl apply ...
HOME=/tmp/cluster-b mkectl upgrade ...

This isolates each cluster's kubeconfig, logs, and state to its own directory, allowing parallel installs and upgrades on the same host. Note that qa-mke already uses this pattern.

Parallel mkectl upgrade invocations interfere with each other#

Running two mkectl upgrade commands concurrently on the same host — even targeting different clusters — causes failures during the pre-upgrade backup step. The root cause is that mkectl writes to hard-coded, non-namespaced paths under the system root temp directory, /tmp on Linux, so concurrent runs collide on such shared paths as /tmp/k0s, /tmp/mke4.yaml, and /tmp/etcdctl.

Workaround:

Export a per-invocation TMPDIR before running the mkectl upgrade command:

TMPDIR=$(mktemp -d) mkectl upgrade ...

This causes the Go os.TempDir() to resolve to an invocation-specific path, eliminating the collision without requiring file locks or serialization. Parallel upgrades can proceed independently. Note that qa-mke already handles this automatically by scoping a tempfile.TemporaryDirectory() to each phase working directory.

Child clusters default to dev registry when registries is omitted#

When a MkeChildConfig is applied without an explicit registries block, a child cluster is deployed using the development registry instead of the production registry.

Workaround:

Always set the registries block explicitly in MkeChildConfig.spec.

apiVersion: mke.mirantis.com/v1alpha1
kind: MkeChildConfig
metadata:
  name: my-mke-child
  namespace: k0rdent
spec:
  registries:
    chartRegistry:
      url: oci://registry.mirantis.com/mke
    imageRegistry:
      url: registry.mirantis.com/mke

Expiring certificates can cause control plane failure#

Cluster certificates may expire a year after install or upgrade from MKE 3, rendering control plane components non-functional.

Workaround:

Restart the k0s controller service on the manager nodes to force certificate recreation. If you have a load balancer for manager nodes, you can manually force cert rotation without any visible downtime.

SSH into each manager node, one by one, and run the following commands:

  1. Remove the etcd server and peer certificates to force renewal:

    rm -f /var/lib/k0s/pki/etcd/server.crt
rm -f /var/lib/k0s/pki/etcd/server.key

  2. Restart the k0s controller service:

    systemctl restart k0scontroller

projectsveltos component can become stuck if it fails to deploy#

If the projectsveltos component fails to deploy when the mkectl apply command is run, for example because of temporary resource pressure on a node or a transient network hiccup, it can become stuck in a failed state from which it cannot recover on its own. When this happens, the command will not complete, even once the original cause has been cleared and mkectl apply is run again. Other cluster components are not affected.

Workaround:

Trigger a fresh reconcile of the projectsveltos release to clear the failed state. To do this, run the following against the cluster:

ts=$(date +%s)
kubectl --kubeconfig ~/.mke/mke.kubeconf -n k0rdent annotate helmrelease projectsveltos \
  reconcile.fluxcd.io/requestedAt="$ts" \
  reconcile.fluxcd.io/resetAt="$ts" \
  --overwrite

Once the release reconciles and reports as ready, mkectl apply will continue and successfully complete.

Dex refresh tokens accumulate for concurrent logins of the same user#

When multiple fresh logins for the same user occur simultaneously — for example, parallel CI/CD jobs sharing a service account and using mkectl ROPC (grant_type=password) — Dex creates a separate refreshtokens.dex.coreos.com object for each login without deleting the previous ones. The race condition leaves the user's single offlinesessions.dex.coreos.com object pointing to only one of those tokens; the rest are orphaned.

Dex does not perform garbage collection for refresh tokens, so orphaned tokens accumulate indefinitely. Over time, this bloats Dex storage and etcd and can degrade performance.

Workaround:

To reduce the likelihood of the race condition, avoid making many simultaneous fresh logins under the same identity. Where possible, reuse the local token cache across pipeline runs so cached clients refresh in place rather than re-authenticate.

If the race condition is already occurring, run the following script to clean up orphaned refresh tokens. The script is non-disruptive, as it keeps the token referenced by each user's current offlinesessions object and deletes only the orphaned duplicates. Adjust the namespace as necessary.

export KUBECONFIG=~/.mke/mke.kubeconf
declare -A live
for id in $(kubectl get offlinesessions.dex.coreos.com -n mke -o json | jq -r '.items[].refresh[].ID'); do
  live[$id]=1
done
for rt in $(kubectl get refreshtokens.dex.coreos.com -n mke -o name | cut -d/ -f2); do
  [[ ${live[$rt]} ]] || kubectl delete refreshtokens.dex.coreos.com -n mke "$rt"
done

Child clusters blocked as a result of Management cluster upgrade#

Child cluster deployments are blocked whenever the management cluster is upgraded to 4.2.0 from MKE 3.9.x or any MKE 4 version. As a result, some of the components that are needed to bootstrap and manage child clusters fail to install during the upgrade process.

Workaround:

Run the mkectl apply command to provision the missing components.

Fresh installation interrupted when the admin kubeconfig is not yet written#

The mkectl apply command for a fresh installation is interrupted during the "Run k0sctl apply" phase, at the point where a k0s controller is up but the admin kubeconfig has yet to be written to ~/.mke/mke.kubeconf. Re-running mkectl apply then fails at the "Collect cluster facts" phase with the following message:

Error: apply failed: phase 'Collect cluster facts' failed: unable to initialize kube clients:
... stat /home/<user>/.mke/mke.kubeconf: no such file or directory

Workaround:

Important

The workaround detailed herein applies only to an interrupted fresh MKE 4.2.0 installation, as there is no running cluster or jeopardized data.

  1. Run the mkectl reset --force command, to tear down the interrupted installation.

  2. Run the mkectl apply -f mke4.yaml command, to re-run the fresh install.

Windows worker nodes retain stale CNI state after reset#

The mkectl reset command intermittently leaves stale Calico Host NetworkingService (HNS) state on a Windows worker node. When you subsequently rejoin the node to a cluster, the Windows Pods scheduled to it remain in the ContainerCreating state and do not receive a Pod IP; the affected Pod's events report hcs::CreateComputeSystem ... The endpoint was not found.

To determine whether a node is affected, list its Calico HNS networks from an elevated PowerShell session. The presence of any such network confirms the issue:

Get-HnsNetwork | Where-Object { $_.Name -like '*calico*' -or $_.Name -eq 'External' }

Workaround:

Run the following in an elevated PowerShell session on the affected node:

  1. If the node has already rejoined a cluster, stop the k0s worker service first, so the removal that follows does not hang. Skip this step on a node that was only reset. 

    Stop-Service k0sworker -Force

  2. Remove the stale Calico HNS endpoints and networks:

    if (-not (Get-Command Get-HnsNetwork -ErrorAction SilentlyContinue)) {
      Import-Module (Join-Path $env:SystemRoot 'System32\WindowsPowerShell\v1.0\Modules\hns\hns.psm1')
}    
Get-HnsEndpoint | Where-Object { $_.Name -like '*calico*' } | Remove-HnsEndpoint -ErrorAction SilentlyContinue    
Get-HnsNetwork  | Where-Object { $_.Name -like '*calico*' -or $_.Name -eq 'External' } | Remove-HnsNetwork -ErrorAction SilentlyContinue

  3. If you stopped the k0s worker service, restart it (Start-Service k0sworker); otherwise run mkectl apply to rejoin the node. Calico subsequently programs a clean HNS network, and the affected Pods will reach the Running state.

Info

If Remove-HnsNetwork hangs, reboot the node with Restart-Computer -Force. On restart, k0s and Calico cleanly reprogram the HNS dataplane.

Root certificate causes LDAP login failure after MKE 3 to MKE 4 upgrade#

When an MKE 3 cluster being upgraded to MKE 4 has an LDAP root certificate configured through the rootCerts parameter, the migration maps the PEM certificate content into the Dex file path spec.authentication.ldap.ca rather than the raw certificate data file path spec.authentication.ldap.caData. As a result, the LDAP connector fails to initialize, and the following error is presented on MKE 4 LDAP login page:

Authentication error

Error type: Bad Request
Error message: Connector failed to initialize

Workaround:

Move the certificate content from ca to caData and clear the old ca value, so the LDAP connector initializes and logins can succeed.

  1. Copy the value from ca to caData:

    export KUBECONFIG=~/.mke/mke.kubeconf
CA=$(kubectl get mkeconfig mke -n mke -o jsonpath='{.spec.authentication.ldap.ca}')
kubectl patch mkeconfig mke -n mke --type merge \
  -p "{\"spec\":{\"authentication\":{\"ldap\":{\"caData\":$(jq -Rs . <<<"$CA")}}}}"

  2. Remove the old ca key:

    export KUBECONFIG=~/.mke/mke.kubeconf
kubectl patch mkeconfig mke -n mke --type merge \
  -p '{"spec":{"authentication":{"ldap":{"ca":null}}}}'

Controller node not added when it is first in the hosts list#

The mkectl apply command fails to add a controller node when that is placed first in the hosts list in the mke4.yaml configuration file. The failure occurs at the Sync Encryption Configuration step with:

✗ Sync Encryption Configuration
  Error: unable to upload file 'encryption.cfg' to host <existing-controller>:
  file /var/lib/k0s/encryption.cfg already exists and differs on node <existing-controller>

Note

The error names an existing controller, not the new node.

Workaround:

When you add or re-add a controller node to an existing cluster, place the host entry for the new controller after the existing controllers on the host list in the mke4.yaml configuration file, then run mkectl apply. Note that the new node can be placed ahead of any worker entries.

When the new controller node is first on the hosts list it causes the sync to treat its empty or freshly generated encryption.cfg file as canonical and attempts to push it onto the healthy controllers, which causes the operation to abort.

Nodes must be rebalanced following restore operation#

Following cluster restore, all of the MKE 4 system components run on the single bootstrap node and are not automatically rebalanced when the remaining control plane nodes rejoin as a result of the mkectl apply command. As such, although all Pods report Running, critical management components remain on the bootstrap node.

Workaround:

To restore High Availability and to enhance node use, once all of the control plane nodes have rejoined the cluster and reported as Ready, force a rollout to reschedule the components across the nodes:

kubectl rollout restart deployment -n mke
kubectl rollout restart deployment -n k0rdent

Info

Rebalancing is not a factor for clusters that have a single controller node.

Namespace-scoped grants migrated from MKE 3 do not inherit team membership#

When you upgrade from MKE 3 to MKE 4, a namespace-scoped Kubernetes grant held by a team or organization is migrated with its members captured at the moment of the upgrade. The migrated grant is not reconciled against team membership afterward: users added to the team after the upgrade do not receive that grant's access in the namespace. Members present at the time of the upgrade are unaffected, and cluster-wide grants are unaffected — they continue to track membership through the team's aggregated role.

In the Grants view, the migrated grant is indistinguishable from a grant created natively in MKE 4. Name, role, and role kind match; only the namespace differs. Nothing in the MKE Dashboard indicates that the grant will fail to extend to new members.

As an example, if organization acme has a team dev granted the admin role in the dev-ns namespace during the upgrade, a user added to dev after the upgrade is denied access to dev-ns, despite the grant appearing valid.

Workaround:

Re-create the affected grant in MKE 4:

  1. In the MKE Dashboard, navigate to Access Control > Grants.
  2. Create a new grant with the same team, role, and namespace as the migrated one.

A grant created in MKE 4 tracks team membership, and thus all current and future members receive access. Repeat this for any team whose membership changes after the upgrade and that held a namespace-scoped grant in MKE 3. The original migrated grant is harmless and additive, and you can delete it once you have re-created the grant.