Enable GPU support using the NVIDIA GPU Operator

MKE provides graphics processing unit (GPU) support for Kubernetes workloads that run on Linux worker nodes. This section describes how to configure MKE and the GPU nodes for the NVIDIA GPU Operator, install the GPU Operator, and run GPU workloads. The GPU Operator installs and manages the GPU driver, the container toolkit, and the device plugin as Kubernetes workloads.

Validated versions

Mirantis validated the following configurations on NVIDIA A10G GPUs using Helm 4.1.3. The RHEL nodes ran on AWS images that use Red Hat Update Infrastructure (RHUI), which the driver container cannot use. Therefore, Mirantis also registered those nodes with subscription-manager, as described in Register RHEL nodes on AWS with Red Hat Subscription Management.

MKE	Kubernetes	Operating system	OS kernel	GPU Operator	NVIDIA driver
3.9.7	1.34.9-mirantis-2	Ubuntu 26.04	7.0.0-1008-aws	v26.7.1	595.91.07
3.9.7	1.34.9-mirantis-2	RHEL 10.2	6.12.0-211.37.1.el10_2	v26.7.1	595.91.07
3.9.7	1.34.9-mirantis-2	RHEL 9.8	5.14.0-687.15.1.el9_8	v26.7.1	595.91.07
3.9.7	1.34.9-mirantis-2	Rocky Linux 9.8	5.14.0-687.15.1.el9_8	v26.7.1	595.91.07
3.9.6	1.34.9-mirantis-2	Ubuntu 24.04	6.8.0-1062-aws	v26.7.1	595.91.07
3.9.6	1.34.9-mirantis-2	Ubuntu 24.04	6.8.0-1062-aws	v24.9.2	580.95.05
3.9.6	1.34.9-mirantis-2	Rocky Linux 9.6	5.14.0-570.58.1.el9_6	v26.7.1	595.91.07
3.8.17	1.31.13-mirantis-2	RHEL 9.8	5.14.0-687.15.1.el9_8	v26.7.1	595.91.07
3.8.17	1.31.13-mirantis-2	RHEL 9.6	5.14.0-687.15.1.el9_8	v26.7.1	595.91.07
3.8.17	1.31.13-mirantis-2	Rocky Linux 9.8	5.14.0-687.15.1.el9_8	v26.7.1	595.91.07
3.8.17	1.31.13-mirantis-2	Rocky Linux 9.6	5.14.0-570.58.1.el9_6	v26.7.1	595.91.07
3.8.16	1.31.13-mirantis-2	Ubuntu 24.04	6.8.0-1062-aws	v26.7.1	595.91.07
3.8.6	1.31.8-mirantis-1	RHEL 8.10	4.18.0-553.158.1.el8_10	v25.10.1	580.95.05

Configure MKE for the GPU Operator

The GPU Operator runs privileged workloads and creates Pods that target specific nodes. MKE restricts both by default. Complete the procedures in this section before you install the GPU Operator.

Enable Kubernetes workloads on GPU nodes

Set the orchestrator type of each GPU node to Kubernetes, as described in Change the node orchestrator.

Allow privileged attributes for the GPU Operator

The GPU Operator components require privileged attributes that MKE withholds from non-administrator subjects. MKE verifies both the account that creates the workload and the service account under which the workload runs, and thus you must allow both. For details on privileged attributes, see Use admission controllers for access.

1. Obtain the current MKE configuration file for your cluster.

2. In the [cluster_config] section of the MKE configuration file, set nvidia_device_plugin to false. The GPU Operator deploys its own device plugin.

3. Add the following four settings to the [cluster_config] section. Add each key even if it is not present in the downloaded file:

   priv_attributes_allowed_for_user_accounts = [
     "hostbindmounts",
     "privileged",
     "hostPID"
   ]
   
   priv_attributes_user_accounts = [
     "gpu-operator:gpu-operator"
   ]
   
   priv_attributes_allowed_for_service_accounts = [
     "hostbindmounts",
     "privileged",
     "hostIPC",
     "hostPID"
   ]
   
   priv_attributes_service_accounts = [
     "gpu-operator:nvidia-gpu-feature-discovery",
     "gpu-operator:nvidia-driver",
     "gpu-operator:nvidia-container-toolkit",
     "gpu-operator:nvidia-operator-validator",
     "gpu-operator:nvidia-device-plugin",
     "gpu-operator:nvidia-dcgm-exporter",
     "gpu-operator:nvidia-mig-manager"
   ]
   

4. Upload the new MKE configuration file.

Allow service accounts to schedule on nodes

The nvidia-operator-validator Pod creates Pods under a service account. Allow service accounts to schedule workloads on all nodes, as described in Allow services deployment on Kubernetes MKE manager or MSR nodes. Enable the option Allow all authenticated users, including service accounts, to schedule on all nodes, including MKE managers and MSR nodes.

Prepare the GPU nodes

Prepare each GPU node before you install the GPU Operator. First, verify the node prerequisites. If the kernel headers check fails, perform the procedure for the operating system of the node.

Prerequisites

1. Verify that the node has an NVIDIA GPU:

   lspci | grep -i nvidia
   

2. Verify that your GPU is a supported NVIDIA GPU product for the driver version that you intend to use.

3. Verify that the host has no NVIDIA driver and that the kernel has not loaded the nouveau module. The GPU Operator installs the driver in a container, and the installation fails if a host driver holds the device:

   lsmod | grep -iE 'nvidia|nouveau'
   

   The command must return no output.

4. Verify that the kernel headers for the running kernel are available to the driver container. The driver container compiles the NVIDIA kernel module on the node and installs the header packages for the running kernel from inside the container. A copy of these packages installed on the host is not visible to the container.

   UbuntuRHELRocky Linux

   The driver container installs linux-headers-$(uname -r). Verify that the package has an installation candidate:

   apt-cache policy linux-headers-$(uname -r)
   

   If the command returns no installation candidate, move the node to a kernel with published headers, as described in Move Ubuntu nodes to a kernel with published headers.

   The driver container installs kernel-devel-$(uname -r) and kernel-headers-$(uname -r) only from the Red Hat Content Delivery Network (CDN) repository that subscription-manager writes, for example, rhel-9-for-x86_64-appstream-rpms. If the node runs an AWS image that uses RHUI, first register the node, as described in Register RHEL nodes on AWS with Red Hat Subscription Management.

   1. Obtain the running kernel version:

   uname -r
   

   Example output:

   6.12.0-211.37.1.el10_2.x86_64
   

   1. Verify that the packages for the running kernel are available. Replace 6.12.0-211.37.1 with the version from uname -r without the distribution and architecture suffix:

   sudo dnf list --showduplicates kernel-devel | grep -F '6.12.0-211.37.1'
   sudo dnf list --showduplicates kernel-headers | grep -F '6.12.0-211.37.1'
   

   Example output on an AWS node registered with subscription-manager:

   kernel-devel.x86_64    6.12.0-211.37.1.el10_2  rhel-10-for-x86_64-appstream-rpms
   kernel-devel.x86_64    6.12.0-211.37.1.el10_2  rhel-10-appstream-rhui-rpms
   kernel-headers.x86_64  6.12.0-211.37.1.el10_2  rhel-10-for-x86_64-appstream-rpms
   kernel-headers.x86_64  6.12.0-211.37.1.el10_2  rhel-10-appstream-rhui-rpms
   

   The output must include the rhel-*-for-x86_64-appstream-rpms repository. The driver container cannot use a package that is available only from rhel-*-appstream-rhui-rpms.

   The driver container installs kernel-devel-$(uname -r) and kernel-headers-$(uname -r). Verify that both packages are available:

   dnf list available "kernel-devel-$(uname -r)" \
   "kernel-headers-$(uname -r)"
   

   On Rocky Linux 9.6, the current repositories no longer publish el9_6 kernel headers. Supply the vault repositories to the driver container, as described in Supply the Rocky Linux 9.6 vault repositories to the driver container.

Move Ubuntu nodes to a kernel with published headers

The Ubuntu archive stops publishing headers for older kernels. If apt-cache policy linux-headers-$(uname -r) returns no installation candidate, the driver container cannot install the headers for the running kernel.

Install a newer kernel for which the archive still publishes headers, and then reboot the node. For example, Mirantis moved the Ubuntu 24.04 validation nodes from the 6.8.0-1018-aws kernel, for which the archive no longer published headers, to the 6.8.0-1062-aws kernel.

Register RHEL nodes on AWS with Red Hat Subscription Management

Note

Perform this procedure only on RHEL GPU nodes that run an AWS image that uses Red Hat Update Infrastructure (RHUI). A node that you already registered with subscription-manager, with manage_repos enabled, does not require it.

An AWS RHEL image receives packages through RHUI. On the host, dnf uses /etc/yum.repos.d/redhat-rhui.repo and the certificates under /etc/pki/rhui, which is sufficient for the host. The driver container does not read that configuration. It installs the headers from the Red Hat CDN repository of a node registered with Red Hat Subscription Management, and it mounts the following host paths:

• /etc/yum.repos.d/redhat.repo
• /etc/pki/entitlement
• /etc/rhsm

AWS RHUI does not create redhat.repo and does not place entitlement certificates in /etc/pki/entitlement.

To register an AWS RHEL node:

1. Register the node with a Red Hat account. The account must have an AppStream entitlement for the RHEL major version of the node:

   sudo subscription-manager register
   

   The command prompts for the Red Hat user name and password and registers the system with subscription.rhsm.redhat.com.

2. Verify the registration:

   sudo subscription-manager identity
   

3. Enable repository management and refresh the subscription data:

   sudo subscription-manager config --rhsm.manage_repos=1
   sudo subscription-manager refresh
   

4. Verify that kernel-devel and kernel-headers for the running kernel are available from the CDN repository, as described in Prerequisites.

5. Repeat the procedure on each AWS RHEL GPU node.

Supply the Rocky Linux 9.6 vault repositories to the driver container

The 595.91.07-rocky9 driver image installs packages with dnf --releasever=9, which selects the current Rocky Linux 9 AppStream. That stream no longer publishes el9_6 kernel headers. The 9.6 packages remain in the Rocky Linux 9.6 vault. The driver container does not read the host yum configuration, so you must supply the vault repository as a ConfigMap.

Note

The validated kernel is 5.14.0-570.58.1.el9_6. An older 9.6 kernel, such as 5.14.0-570.17.1.el9_6, has no matching kernel-devel in that vault.

1. Add the 9.6 vault repositories:

   sudo tee /etc/yum.repos.d/rocky-9.6-vault.repo <<'EOF'
   [vault-9.6-baseos]
   name=Rocky 9.6 vault BaseOS
   baseurl=https://dl.rockylinux.org/vault/rocky/9.6/BaseOS/$basearch/os/
   gpgcheck=1
   enabled=1
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
   
   [vault-9.6-appstream]
   name=Rocky 9.6 vault AppStream
   baseurl=https://dl.rockylinux.org/vault/rocky/9.6/AppStream/$basearch/os/
   gpgcheck=1
   enabled=1
   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
   EOF
   

2. Verify that the vault publishes headers for the running kernel:

   sudo dnf clean metadata
   dnf list available "kernel-devel-$(uname -r)" \
   --disablerepo='*' --enablerepo=vault-9.6-appstream --enablerepo=vault-9.6-baseos
   

3. If the package is absent, install the newest el9_6 kernel, headers, and development package that the vault lists, and then reboot the node. The following commands install the validated set:

   sudo dnf install -y --disablerepo='*' \
   --enablerepo=vault-9.6-appstream --enablerepo=vault-9.6-baseos \
   kernel-5.14.0-570.58.1.el9_6 \
   kernel-devel-5.14.0-570.58.1.el9_6 \
   kernel-headers-5.14.0-570.58.1.el9_6
   
   sudo reboot
   

   Caution

   Leave the rest of the system on Rocky Linux 9.6. Upgrading the system moves the node to a newer minor release, for which this procedure does not apply.

4. Create a ConfigMap from the vault repository file. The GPU Operator uses the ConfigMap name as a volume name, and a volume name cannot contain a dot, so use rocky96-vault-repos:

   kubectl create namespace gpu-operator
   
   kubectl create configmap rocky96-vault-repos -n gpu-operator \
   --from-file=rocky-9.6-vault.repo=/etc/yum.repos.d/rocky-9.6-vault.repo
   

   Note

   Run the kubectl commands from a host that has the repository file. If you run them from an administrator workstation, copy rocky-9.6-vault.repo to that workstation first and pass its local path to --from-file.

When you install the GPU Operator, set driver.repoConfig.configMapName=rocky96-vault-repos, as described in Install the Helm chart.

Install the GPU Operator

Install the GPU Operator Helm chart with the settings that match your GPU Operator version.

Install the Helm chart

1. Optional. Verify that a driver container image exists for your driver version and operating system. The GPU Operator composes the image tag from the driver version that you specify and does not validate it:

   TOKEN=$(curl -s "https://nvcr.io/proxy_auth?scope=repository:nvidia/driver:pull" | jq -r .token)
   curl -s -H "Authorization: Bearer $TOKEN" \
   https://nvcr.io/v2/nvidia/driver/tags/list | jq -r '.tags[]'
   

   Verify that a <driver-version>-<operating-system> tag is present, for example, 595.91.07-ubuntu24.04 or 595.91.07-ubuntu26.04.

2. Add the NVIDIA Helm repository:

   helm repo add nvidia https://nvidia.github.io/gpu-operator
   helm repo update
   

3. Optional. Exclude the nodes that must not run GPU workloads. The GPU Operator selects every node on which it detects an NVIDIA device, including manager nodes:

   kubectl label node <node-name> nvidia.com/gpu.deploy.operands=false --overwrite
   

4. Install the chart:

   helm install gpu-operator nvidia/gpu-operator \
   -n gpu-operator --create-namespace \
   --version v26.7.1 \
   --set nfd.enabled=false \
   --set driver.version=595.91.07 \
   --set operator.runtimeClass=docker \
   --set cdi.enabled=false \
   --set-json 'toolkit.env=[{"name":"NVIDIA_RUNTIME_SET_AS_DEFAULT","value":"true"}]'
   

   • nfd.enabled=false prevents the chart from installing a second Node Feature Discovery deployment. MKE already runs Node Feature Discovery, and installation of the chart copy fails on CustomResourceDefinition (CRD) ownership.
   • driver.version=595.91.07 is optional. The GPU Operator installs and manages the driver either way.
   • Whether to set operator.runtimeClass, cdi.enabled, and NVIDIA_RUNTIME_SET_AS_DEFAULT depends on the GPU Operator version. For details, see Select RuntimeClass settings for your GPU Operator version.
   • On Rocky Linux 9.6, also add --set driver.repoConfig.configMapName=rocky96-vault-repos to supply the vault repositories that you configured in Supply the Rocky Linux 9.6 vault repositories to the driver container.

   Installation of the driver takes several minutes, because the driver container compiles the kernel module on each GPU node.

Select RuntimeClass settings for your GPU Operator version

The required chart settings depend on the GPU Operator version.

GPU Operator versions earlier than v25.10.0, for example, v24.9.2, do not create a RuntimeClass. Omit operator.runtimeClass, cdi.enabled, and NVIDIA_RUNTIME_SET_AS_DEFAULT:

helm install gpu-operator nvidia/gpu-operator \
-n gpu-operator --create-namespace \
--version v24.9.2 \
--set nfd.enabled=false \
--set driver.version=580.95.05

GPU Operator v25.10.0 and later, including the validated v25.10.1 and v26.7.1, require operator.runtimeClass=docker. These versions create a RuntimeClass named nvidia and set runtimeClassName: nvidia on the operand Pods. MKE cri-dockerd accepts only an empty handler or the docker handler. Pods that request nvidia remain in the Init state with no IP address:

Failed to create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "nvidia" not supported

Install GPU Operator v25.10.0 and later with the following settings, as in the v26.7.1 command in Install the Helm chart:

• operator.runtimeClass=docker creates a single RuntimeClass named docker, with the docker handler, instead of the default nvidia handler.
• cdi.enabled=false prevents the GPU Operator from creating the additional docker-cdi and docker-legacy RuntimeClasses, the handlers of which cri-dockerd rejects.
• toolkit.env=[{"name":"NVIDIA_RUNTIME_SET_AS_DEFAULT","value":"true"}] makes the NVIDIA runtime the default Docker runtime on each GPU node, and thus your GPU workloads do not need to set runtimeClassName.

Verify the installation

1. On GPU Operator v25.10.0 and later, verify that exactly one RuntimeClass exists and that the GPU Operator applies it to the operands. GPU Operator v24.9.2 does not create a RuntimeClass, so kubectl get runtimeclass returns no objects on that version.

   kubectl get runtimeclass
   
   kubectl get ds -n gpu-operator \
   -o custom-columns='DS:.metadata.name,RUNTIMECLASS:.spec.template.spec.runtimeClassName'
   

   Example output:

   NAME     HANDLER   AGE
   docker   docker    4m
   
   DS                                   RUNTIMECLASS
   gpu-feature-discovery                docker
   nvidia-container-toolkit-daemonset   <none>
   nvidia-dcgm-exporter                 docker
   nvidia-device-plugin-daemonset       docker
   nvidia-driver-daemonset              <none>
   nvidia-operator-validator            docker
   

   The <none> value for the driver and container toolkit DaemonSets is expected, because these components are privileged and mount the devices themselves.

2. Verify that all GPU Operator components are running and that each Pod has an IP address. The components may take a few minutes to start.

   kubectl get pods -n gpu-operator -o wide
   

   Example output:

   NAME                                       READY   STATUS      RESTARTS   AGE     IP                NODE                         NOMINATED NODE   READINESS GATES
   gpu-feature-discovery-224d8                1/1     Running     0          5m52s   192.168.232.136   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   gpu-feature-discovery-62jfm                1/1     Running     0          5m52s   192.168.85.199    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   gpu-operator-5946c45d5d-4pmzk              1/1     Running     0          6m5s    192.168.232.130   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   nvidia-container-toolkit-daemonset-5zdqn   1/1     Running     0          5m54s   192.168.232.132   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   nvidia-container-toolkit-daemonset-t29dd   1/1     Running     0          5m54s   192.168.85.195    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   nvidia-cuda-validator-pf7m5                0/1     Completed   0          2m42s   192.168.232.137   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   nvidia-cuda-validator-v7jj5                0/1     Completed   0          2m33s   192.168.85.200    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   nvidia-dcgm-exporter-255h6                 1/1     Running     0          5m52s   192.168.85.198    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   nvidia-dcgm-exporter-gz7zv                 1/1     Running     0          5m52s   192.168.232.135   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   nvidia-device-plugin-daemonset-ph2nn       1/1     Running     0          5m53s   192.168.232.134   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   nvidia-device-plugin-daemonset-r2ctz       1/1     Running     0          5m53s   192.168.85.197    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   nvidia-driver-daemonset-9mq2z              1/1     Running     0          5m54s   192.168.232.131   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   nvidia-driver-daemonset-tjfhz              1/1     Running     0          5m54s   192.168.85.194    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   nvidia-operator-validator-jlr6b            1/1     Running     0          5m53s   192.168.85.196    gpu-test-ubuntu26-ubuntu-1   <none>           <none>
   nvidia-operator-validator-nd7gf            1/1     Running     0          5m53s   192.168.232.133   gpu-test-ubuntu26-ubuntu-2   <none>           <none>
   

   The nvidia-cuda-validator Pods report Completed, which is their successful terminal state. All other Pods report 1/1 Running. A Pod without an IP address indicates that the container runtime refused the sandbox.

3. Verify the driver version:

   kubectl exec -n gpu-operator ds/nvidia-driver-daemonset \
   -c nvidia-driver-ctr -- nvidia-smi
   

   Example output:

   +-----------------------------------------------------------------------------------------+
   | NVIDIA-SMI 595.91.07              Driver Version: 595.91.07      CUDA Version: 13.2     |
   +-----------------------------------------+------------------------+----------------------+
   | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
   |                                         |                        |               MIG M. |
   |=========================================+========================+======================|
   |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
   |  0%   30C    P8             25W /  300W |       0MiB /  23028MiB |      0%      Default |
   |                                         |                        |                  N/A |
   +-----------------------------------------+------------------------+----------------------+
   
   +-----------------------------------------------------------------------------------------+
   | Processes:                                                                              |
   |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
   |        ID   ID                                                               Usage      |
   |=========================================================================================|
   |  No running processes found                                                             |
   +-----------------------------------------------------------------------------------------+
   

4. Verify that the nodes advertise the resource and that the count matches the number of physical GPUs on each node:

   kubectl get nodes \
   -o custom-columns='NODE:.metadata.name,GPU:.status.capacity.nvidia\.com/gpu'
   

   Example output:

   NODE       GPU
   manager0   <none>
   worker0    1
   worker1    1
   

Schedule GPU workloads

The following example describes how to deploy a simple workload that reports detected NVIDIA CUDA devices.

1. Create a practice Deployment that requests nvidia.com/gpu in the limits section. Kubernetes schedules the Pod on any available GPU in your system.

   kubectl apply -f- <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       run: gpu-test
     name: gpu-test
   spec:
     replicas: 1
     selector:
       matchLabels:
         run: gpu-test
     template:
       metadata:
         labels:
           run: gpu-test
       spec:
         containers:
         - command:
           - sh
           - -c
           - "/cuda-samples/sample && sleep infinity"
           image: nvcr.io/nvidia/k8s/cuda-sample:devicequery-cuda12.5.0-ubuntu22.04
           name: gpu-test
           resources:
             limits:
               nvidia.com/gpu: "1"
   EOF
   

   Note

   The Pod does not set runtimeClassName. Because the NVIDIA runtime is the default Docker runtime on GPU nodes, GPU workloads require no runtime configuration of their own.

2. Verify that the Pod is in the Running state:

   kubectl get pods | grep "gpu-test"
   

   Example output:

   NAME                        READY   STATUS    RESTARTS   AGE
   gpu-test-6d4f9c7b85-hpv74   1/1     Running   0          14m
   

3. Review the logs. The presence of Result = PASS indicates a successful deployment:

   kubectl logs <pod-name>
   

   Example output:

   /cuda-samples/sample Starting...
   
    CUDA Device Query (Runtime API) version (CUDART static linking)
   
   Detected 1 CUDA Capable device(s)
   
   Device 0: "NVIDIA A10G"
   ...
   
   deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.2, CUDA Runtime Version = 12.5, NumDevs = 1
   Result = PASS
   

4. Determine the overall GPU capacity of your cluster by inspecting its nodes:

   kubectl get nodes \
   -o jsonpath='{range .items[*]}{.status.capacity.nvidia\.com/gpu}{"\n"}{end}' \
   | awk '{s+=$1} END {print s+0}'
   

5. Scale the Deployment to the number of available GPUs:

   kubectl scale deployment/gpu-test --replicas N
   

6. Verify that Kubernetes scheduled all of the replicas:

   kubectl get pods | grep "gpu-test"
   

   Example output:

   NAME                        READY   STATUS    RESTARTS   AGE
   gpu-test-6d4f9c7b85-hpv74   1/1     Running   0          12m
   gpu-test-6d4f9c7b85-swrrx   1/1     Running   0          11m
   

7. Remove the Deployment and corresponding Pods:

   kubectl delete deployment gpu-test
   

Troubleshooting

Operand Pods have no IP address and remain in the Init state

This issue occurs on GPU Operator v25.10.0 and later when you leave operator.runtimeClass at the chart default, nvidia. GPU Operator v24.9.2 does not create a RuntimeClass and does not produce this issue.

The container runtime refused the Pod sandbox because the Pod requests a RuntimeClass handler that cri-dockerd does not recognize. Review the Pod events:

kubectl describe pod -n gpu-operator <pod-name>

Example output:

Warning  FailedCreatePodSandBox  12s (x15 over 3m1s)  kubelet  Failed to create pod sandbox:
rpc error: code = Unknown desc = RuntimeHandler "nvidia" not supported

Verify that you installed the v25.10.0 or later chart with the operator.runtimeClass=docker and cdi.enabled=false settings, as described in Select RuntimeClass settings for your GPU Operator version. GPU Operator v24.9.2 does not take those settings.

Pods that started before the container toolkit configured the NVIDIA runtime also fail in this way and do not recover. Delete them so that their controllers recreate them:

kubectl delete pod -n gpu-operator -l app=nvidia-operator-validator

The driver Pod remains in the Pending state on an AWS RHEL node

The node has no /etc/yum.repos.d/redhat.repo file, which the driver container mounts from the host. AWS RHEL images that use RHUI do not create this file. Review the Pod events:

kubectl describe pod -n gpu-operator <driver-pod-name>

The events contain the following message:

hostPath type check failed: /etc/yum.repos.d/redhat.repo is not a file

Register the node with Red Hat Subscription Management, as described in Register RHEL nodes on AWS with Red Hat Subscription Management.

Caution

Do not create an empty redhat.repo file instead. It allows the Pod to start, but the driver container then exits, because the repositories built into the driver image do not publish kernel-devel for the node kernel.

The driver container exits a few seconds after it starts

The kernel headers for the running kernel are not available from the distribution repositories. Review the driver container logs:

kubectl logs -n gpu-operator <driver-pod-name> -c nvidia-driver-ctr

Example output on Ubuntu:

Resolving Linux kernel version...
Could not resolve Linux kernel version

Example output on Rocky Linux and RHEL:

Installing Linux kernel headers...
Error: Unable to find a match: kernel-devel-<kernel> kernel-headers-<kernel>

Make the headers available to the driver container, depending on the operating system of the node:

• Ubuntu: Move Ubuntu nodes to a kernel with published headers.
• RHEL on an AWS image that uses RHUI: Register RHEL nodes on AWS with Red Hat Subscription Management. The host can list kernel-devel from rhel-*-appstream-rhui-rpms, but the driver container cannot use that repository. Until redhat.repo contains the CDN AppStream repository, the container installs packages only from the Red Hat Universal Base Image (UBI) repositories in the driver image, which do not publish kernel-devel.
• Rocky Linux 9.6: Supply the Rocky Linux 9.6 vault repositories to the driver container.

Installing kernel-devel or linux-headers on the host does not resolve the issue, because the driver container installs the package inside the container.

Kubernetes rejects the driver DaemonSet because the repository ConfigMap name contains a dot

The GPU Operator copies driver.repoConfig.configMapName into the driver Pod as a volume name. Kubernetes rejects a volume name that contains a dot:

DaemonSet.apps "nvidia-driver-daemonset" is invalid:
spec.template.spec.volumes[15].name: Invalid value: "rocky-9.6-vault-repos":
must not contain dots

Recreate the ConfigMap as rocky96-vault-repos and set driver.repoConfig.configMapName to that name. The GPU Operator then creates the driver DaemonSet.

The driver container fails to pull its image

No driver container image exists for the combination of the driver version and operating system. Review the Pod events:

kubectl describe pod -n gpu-operator <driver-pod-name>

Example output:

Warning  Failed  41s (x5 over 3m27s)  kubelet  Failed to pull image
"nvcr.io/nvidia/driver:595.104.02-ubuntu24.04": manifest unknown

Select a driver version for which NVIDIA publishes an image, as described in Install the Helm chart.

The nvidia-operator-validator Pod remains in the Init:CrashLoopBackOff state

The nvidia-operator-validator Pod cannot create the Pod with which it validates CUDA, because its service account has no node scheduling access. Review the logs of the cuda-validation container:

kubectl logs -n gpu-operator <validator-pod-name> -c cuda-validation

Example output:

error validating cuda workload: failed to create cuda validation pod,
err pods "nvidia-cuda-validator-" is forbidden: user or Pod Service Account
does not have access to schedule workloads on node <node-name>

Grant node scheduling access to service accounts, as described in Allow service accounts to schedule on nodes.

Additional replicas remain in the Pending state

If you scale the example Deployment beyond the GPU capacity of the cluster, the additional replicas remain in the Pending state with a FailedScheduling event and the Insufficient nvidia.com/gpu message.

1. Add an additional replica:

   kubectl scale deployment/gpu-test --replicas N+1
   kubectl get pods | grep "gpu-test"
   

   Example output:

   NAME                        READY   STATUS    RESTARTS   AGE
   gpu-test-6d4f9c7b85-hpv74   1/1     Running   0          14m
   gpu-test-6d4f9c7b85-swrrx   1/1     Running   0          13m
   gpu-test-6d4f9c7b85-zgwfh   0/1     Pending   0          3m26s
   

2. Review the events of the pending Pod:

   kubectl describe pod gpu-test-6d4f9c7b85-zgwfh
   

   Example output:

   Events:
   Type     Reason            Age        From               Message
   ----     ------            ----       ----               -------
   Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 2 Insufficient nvidia.com/gpu.
   

See also

NVIDIA GPU documentation