Known issues

MKE 3.7.11 known issues with available workaround solutions include:

[MKE-11535][FIELD-7110] ucp-nvidia-gpu-feature-discovery pods may enter CrashLoopBackOff state

Due to the upstream dependency issue in gpu-feature-discovery software, customers may encounter nvidia-gpu-feature-discovery in CrashLoopBackOff state with the following errors:

"--mig-strategy=mixed": executable file not found in $PATH: unknown

I0726 08:42:14.338857       1 main.go:144] Start running
SIGSEGV: segmentation violation
PC=0x0 m=4 sigcode=1
signal arrived during cgo execution

goroutine 1 [syscall]:
runtime.cgocall(0x12f4f20, 0xc00025b720)
 /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc00025b6f8 sp=0xc00025b6c0 pc=0x409b2b

Workaround:

  1. Disable nvidia_device_plugin. Refer to Use an MKE configuration file.

  2. Install NVIDIA GPU components with the NVIDIA GPU Operator.

[MKE-11531] NodeLocal DNS Pods attempt to deploy to Windows nodes

The DNS caching service that NodeLocalDNS deploys to nodes as Pods is a Linux-only solution, however it attempts without success to also deploy to Windows nodes.

Workaround:

  1. Edit the node-local-dns daemonset:

    kubectl edit daemonset node-local-dns -n kube-system

  2. Add the following under spec.template.spec:

    nodeSelector:
  kubernetes.io/os: linux

  3. Save the daemonset.

[MKE-11525] Kubelet node profiles fail to supersede global setting

Flags specified in the global custom_kubelet_flags setting and then applied through kubelet node profiles end up being applied twice.

Workaround:

Do not define any global flags in the global custom_kubelet_flags setting that will be used in kubelet node profiles.

[FIELD-7023] Air-gapped upgrades fail if images are inaccessible

In air-gapped environments, upgrades fail if the MKE images are not preloaded on the selected manager node or the node cannot automatically pull the required MKE images. This results in a rollback to the previous MKE version, which in this particular scenario can inadvertently remove the etcd/RethinkDB cluster from the MKE cluster and thus require you to restore MKE from a backup.

Workaround:

Ensure either that the manager nodes have all necessary MKE images preloaded before performing an upgrade or that they can pull the images from a remote repository.

[MKE-10152] Upgrading large Windows clusters can initiate a rollback

Upgrades can rollback on a cluster with a large number of Windows worker nodes.

Workaround:

Invoke the --manual-worker-upgrade option and then manually upgrade the workers.

[MKE-9699] Ingress Controller with external load balancer can enter crashloop

Due to the upstream Kubernetes issue 73140, rapid toggling of the Ingress Controller with an external load balancer in use can cause the resource to become stuck in a crashloop.

Workaround:

  1. Log in to the MKE web UI as an administrator.

  2. In the left-side navigation panel, navigate to <user name> > Admin Settings > Ingress.

  3. Click the Kubernetes tab to display the HTTP Ingress Controller for Kubernetes pane.

  4. Toggle the HTTP Ingress Controller for Kubernetes enabled control to the left to disable the Ingress Controller.

  5. Use the CLI to delete the Ingress Controller resources:

    kubectl delete service ingress-nginx-controller-admission --namespace ingress-nginx
kubectl delete deployment ingress-nginx-controller --namespace
ingress-nginx

  6. Verify the successful deletion of the resources:

    kubectl get all --namespace ingress-nginx

    Example output:

    No resources found in ingress-nginx namespace.

  7. Return to the HTTP Ingress Controller for Kubernetes pane in the MKE web UI and change the nodeport numbers for HTTP Port, HTTPS Port and TCP Port.

  8. Toggle the HTTP Ingress Controller for Kubernetes enabled control to the right to re-enable the Ingress Controller.

[MKE-8662] Swarm only manager nodes are labeled as mixed mode

When MKE is installed in swarm only mode, manager nodes start off in mixed mode. As Kubernetes installation is skipped altogether, however, they should be labeled as swarm mode.

Workaround: Change the labels following installation.

Change the labels following installation.

[MKE-8914] Windows Server Core with Containers images incompatible with GCP

The use of Windows ServerCore with Containers images will prevent kubelet from starting up, as these images are not compatible with GCP.

As a workaround, use Windows Server or Windows Server Core images.

[MKE-8814] Mismatched MTU values cause Swarm overlay network issues on GCP

Communication between GCP VPCs and Docker networks that use Swarm overlay networks will fail if their MTU values are not manually aligned. By default, the MTU value for GCP VPCs is 1460, while the default MTU value for Docker networks is 1500.

Workaround:

Select from the following options:

  • Create a new VPC and set the MTU value to 1500.

  • Set the MTU value of the existing VPC to 1500.

For more information, refer to the Google Cloud Platform documentation, Change the MTU setting of a VPC network.