NVIDIA GPU Operator¶

Run the real GPU Operator against Mokka: the device plugin, GPU Feature Discovery, DCGM and the validator all come up and behave as they would on a node with hardware.

Why it needs an overlay¶

The GPU Operator's job is to install and manage a driver. Mokka has already provided one, so the operator has to be told to stop at the parts that consume a driver rather than install one — and pointed at where Mokka staged it.

That is what the values below do. Installing the operator with its defaults against a Mokka node fails: the driver DaemonSet tries to build a kernel module that cannot exist.

Prerequisites¶

Takes about 15 minutes, most of it pulling operator images.

Step 1 — Create a cluster with CDI enabled¶

The operator resolves GPUs through the Container Device Interface, so containerd needs CDI turned on and the NVIDIA container toolkit present.

kind create cluster --name mokka-operator

NODE=mokka-operator-control-plane

docker exec "$NODE" bash -c '
  apt-get update -qq && apt-get install -y -qq curl gpg
  curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
  curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed "s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g" \
    > /etc/apt/sources.list.d/nvidia-container-toolkit.list
  apt-get update -qq && apt-get install -y -qq nvidia-container-toolkit
'

docker exec "$NODE" nvidia-ctk runtime configure \
  --runtime=containerd --cdi.enabled --set-as-default
docker exec "$NODE" sed -i 's/^mode = "auto"$/mode = "cdi"/' \
  /etc/nvidia-container-runtime/config.toml
docker exec "$NODE" grep -q '^mode = "cdi"$' \
  /etc/nvidia-container-runtime/config.toml
docker exec "$NODE" systemctl restart containerd

Step 2 — Install Mokka¶

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --wait --timeout 120s

Step 3 — Install the GPU Operator¶

cat > gpu-operator-values.yaml <<'EOF'
driver:
  enabled: false
toolkit:
  enabled: false
dcgm:
  enabled: false
mig:
  strategy: none
migManager:
  enabled: false
nodeStatusExporter:
  enabled: false

cdi:
  enabled: true
  default: true

devicePlugin:
  enabled: true
  config:
    name: ""

gfd:
  enabled: true
  env:
    - name: GFD_MACHINE_TYPE_FILE
      value: "/etc/nvml-mock/machine-type"

dcgmExporter:
  enabled: true
  env:
    - name: DCGM_EXPORTER_COLLECT_INTERVAL
      value: "5000"

validator:
  driver:
    env:
      - name: DISABLE_DEV_CHAR_SYMLINK_CREATION
        value: "true"
  toolkit:
    env:
      - name: NVIDIA_VISIBLE_DEVICES
        value: "all"
  cuda:
    env:
      - name: WITH_WORKLOAD
        value: "false"
EOF

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --version v26.3.3 \
  -f gpu-operator-values.yaml \
  --wait --timeout 600s

The version is deliberate. GPU Operator v26.3.3 is the release CI validates for this workflow. Later releases can add hardware checks or change the chart schema, so validate them against Mokka before updating the pin.

Use a values file, not --set

Every operand takes its driver root through a nested env list, and that shape does not survive the command line: chained --set gfd.env[0].name=… flags drop values silently, --set validator.cuda.env[0].value="false" is typed as a bool against a string field, and an unquoted env[0] is glob-expanded by zsh before Helm sees it. A values file is the only form of this configuration that works.

Step 4 — Verify¶

kubectl -n gpu-operator wait --for=condition=ready pod --all --timeout=300s

kubectl get node "$NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
# 4 — the gb300 default profile has four devices

kubectl get node "$NODE" -o json \
  | jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com")))'

The validator pod reaching Completed is the meaningful signal: it probes the driver root, exercises CDI injection, and checks the node advertises GPUs.

What each value is doing¶

Value	Why
`driver.enabled: false`	Mokka is the driver. The real DaemonSet would try to build a kernel module
`toolkit.enabled: false`	The mock libraries are staged on the host by Mokka's DaemonSet, so the toolkit has nothing to inject. CDI carries the devices instead
`cdi.enabled` / `cdi.default`	The runtime reads `/var/run/cdi/nvidia.yaml`, which Mokka generates. This is what replaces the toolkit operand
`dcgm.enabled: false`	The separate nv-hostengine DaemonSet is redundant: `dcgm-exporter` embeds the host engine in-process
`dcgmExporter.enabled: true`	Kept on deliberately — it reads the mock through libdcgm, which is part of what this proves
`GFD_MACHINE_TYPE_FILE`	GFD's default reads `/sys/class/dmi/id/product_name`, which says `kind` here and is absent on hosts with no DMI. Mokka writes a file of its own
`mig.strategy: none`	The install above does not partition the board, so there are no slices to advertise and whole GPUs are the right view. MIG itself is simulated: MIG partitioning carves a board and serves the slices through the device plugin in `migStrategy=single`
`validator.cuda.WITH_WORKLOAD: false`	The CUDA validation step launches a kernel, and CUDA is not simulated
`DISABLE_DEV_CHAR_SYMLINK_CREATION`	The `/dev/char` symlink step runs `modprobe nvidia`, which cannot work in a Kind container. Mokka already staged those nodes

Three labels that look wrong and are not¶

Inspecting the node labels after a run turns up three that seem to contradict the overlay. None of them breaks anything.

nvidia.com/gpu.deploy.driver=true and nvidia.com/gpu.deploy.container-toolkit=true are the operator's own scheduling hints, written by GFD for operands it manages. They mean "this node is eligible for that operand", not "that operand is here". With driver.enabled=false and toolkit.enabled=false the DaemonSets are never created, so the labels have nothing to select. Confirm with kubectl -n gpu-operator get ds, which lists neither.
nvidia.com/mig.capable=true looks like it contradicts mig.strategy: none, and does not. The board really is MIG-capable and the mock really partitions it; this install simply does not carve it, which is why the strategy stays none. Carving it is the MIG partitioning guide.

Troubleshooting¶

The validator crash-loops. It is a statically linked Go binary with no shell, so a missing file in the staged driver root looks like a crash rather than an error. Check what is actually there:

docker exec "$NODE" ls -la /run/nvidia/driver/usr/lib64/libnvidia-ml.so*

The device plugin reports 0 GPUs. Almost always driver.enabled or toolkit.enabled left at their defaults — the operator is managing a driver that does not exist. Confirm with helm get values gpu-operator.

GFD labels the node kind. GFD_MACHINE_TYPE_FILE is missing from the overlay.

More symptoms in Troubleshooting.

Running it from a checkout¶

The steps above are the whole scenario and never need the repository. If you have it cloned, run.sh performs the same install and then asserts the result:

cd docs/guides/with-gpu-operator && ./run.sh

It installs Mokka, checks that nvidia-smi -L works inside a mock pod, installs the operator with the same overlay from gpu-operator-values.yaml, then asserts two things that can only hold if the real operands read the mock: some node lists nvidia.com/gpu as allocatable, and some node carries a nvidia.com/gpu.product label. Both scan every node, because the operands do not tolerate the control-plane NoSchedule taint.

Variable	Default	Effect
`GPU_PROFILE`	`gb300`	Any profile under the chart's `profiles/` directory
`NAMESPACE`	`mokka-operator`	Namespace for the Mokka release
`OPERATOR_NAMESPACE`	`gpu-operator`	Namespace for the GPU Operator release
`GPU_OPERATOR_VERSION`	`v26.3.3`	GPU Operator chart version to install
`NVML_MOCK_IMAGE`	`ghcr.io/nvidia/nvml-mock:latest`	Published image to install
`BUILD_LOCAL`	`false`	Build the image from source and side-load it with `kind load`
`HELM_TIMEOUT`	`15m`	Wait budget for each Helm install
`DEMO_ASSUME_YES`	`false`	Skip the confirmation prompt and the co-location refusals

One Mokka release per cluster at a time

The chart's hostPath mounts are fixed and release-independent (/var/lib/nvml-mock, /var/run/cdi, /run/nvidia, and the NFD features directory), and it tolerates every taint, so two Mokka releases land on every node and write the same per-node state whatever they are called — including the shared config that any real GPU workload on those nodes then reads. run.sh refuses with exit 4 if another Mokka release is already installed anywhere on the cluster.

Clean up¶

kind delete cluster --name mokka-operator

To read about	See
Every chart value	Installation
Changing GPU state while the operator watches	Runtime Control
Driving this from CI	Use in CI/CD
The full health loop on top of this stack	NVSentinel