nvml-mock Helm Chart

Mock GPU infrastructure for Kubernetes testing. Turns any cluster into a multi-GPU environment using a CGo-based mock NVML library — no physical NVIDIA hardware required.

What It Does

Deploys a DaemonSet that creates on every node: - Mock libnvidia-ml.so shared library at /var/lib/nvml-mock/driver/usr/lib64/ - Mock device nodes at /var/lib/nvml-mock/driver/dev/nvidia{N,ctl,-uvm,-uvm-tools} (consumers see them at /dev/nvidia* via CDI bind-mount) - GPU configuration at /var/lib/nvml-mock/driver/config/config.yaml - An NFD feature file at /etc/kubernetes/node-feature-discovery/features.d/nvml-mock.features, which NFD turns into the node label feature.node.kubernetes.io/pci-10de.present=true (see Node Labels) - A fake InfiniBand sysfs tree at /var/lib/nvml-mock/ib/sys/class/infiniband/... paired with libibmocksys.so (LD_PRELOAD) so real ibstat, ibstatus, iblinkinfo, ... read mock HCAs - A fake PCI sysfs tree at /var/lib/nvml-mock/sys/bus/pci/devices/... (symlinks into /var/lib/nvml-mock/sys/devices/pciDDDD:BB/...) so consumers resolve the PCIe root complex via a standard readlink(). lspci and anything else reaching it through libc read it via libmockfs.so; containers served the nvidia.com/gpu CDI spec get it bind-mounted at the kernel paths, which is what Go consumers need — see PCI sysfs in containers - A fake kernel-module surface at /var/lib/nvml-mock/proc/modules and /var/lib/nvml-mock/sys/module/.... /sys/module/nvidia/refcnt exists, and lsmod lists nvidia, nvidia_uvm, nvidia_modeset, gdrdrv and nvidia_fs, plus nvidia_peermem and mlx5_core when InfiniBand is enabled. That covers every module the GPU Operator validator greps for and the nvidia refcount it stats. The node's own modules are mirrored beside them. libmockfs.so redirects both paths for libc consumers. Both CDI specs bind-mount the tree for Go consumers. A state reconcile refreshes the mirror, so it can lag a module load or unload - The driver's per-GPU procfs entries at /var/lib/nvml-mock/driver/proc/driver/nvidia/gpus/<bdf>/information (model, UUID, bus location and device minor), and each GPU bound to the nvidia driver at /var/lib/nvml-mock/sys/bus/pci/drivers/nvidia/<bdf> with its local_cpulist. The CPU list starts from the profile's CPU affinity for the GPU and keeps only the CPUs the node has online; when none of them exist, it lists every online CPU, as the kernel does for a device with no locality. NVML reports the same list as the GPU's CPU affinity. libmockfs.so serves both at the kernel paths, which is what Slurm's AutoDetect=nvidia reads — see the Slinky guide

Consumers (DRA driver, device plugin) point at /var/lib/nvml-mock/driver as the NVIDIA driver root and discover GPUs through standard NVML APIs.

By default (nri.enabled=true), the chart adds nvml-mock-nri to the node DaemonSet. This node-local containerd NRI plugin mounts the host overlay into newly created containers at /opt/nvml-mock and injects the mock environment at runtime, so a pod needs no Mokka-specific pod spec changes. It injects only containers that hold a GPU allocation from the device plugin or the NVIDIA DRA driver, or whose pod carries the nvml-mock.nvidia.com/devices, nvml-mock.nvidia.com/infiniband or nvml-mock.nvidia.com/imex-channels annotation set to "true". A pod that requested no GPU and carries none of them is left untouched and sees no GPUs, as on a real GPU node. See Which containers are injected for the full rules. Set up NRI injection covers the containerd prerequisite, including on Kind, how to verify the plugin, and how to turn it off with nri.enabled=false.

Install it into its own namespace, and pass -n:

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  -n mokka --create-namespace

The plugin always excludes its own release namespace, so that the main nvml-mock DaemonSet is never self-injected. Install without -n and the release namespace is default — the plugin then renders --excluded-namespaces=default,kube-system and skips every pod a first-time user runs. Nothing reports this: the node DaemonSet is Ready, /readyz returns 200 because the plugin is registered, and skipped containers produce no log line at any level. The pods simply start with no mock GPU.

Prerequisites

Tool	Version	Required For
Docker	20.10+	Building the image
Kind	0.20+	Local cluster (or use your own)
kubectl	1.31+	Cluster access
Helm	3.x	Chart installation
Go	1.25+	Building from source
jq	any	DRA verification only

Published image: The nvml-mock container image is published at ghcr.io/nvidia/nvml-mock, tagged with the release version on every release, and with the chart appVersion (the next -dev version, such as 0.5.0-dev) on pushes to main. The chart installs the image tagged with its appVersion by default, so a released chart pulls its own release and the chart on main pulls the latest main build. If the image is not yet available, use "Option B: Build from source" in the quick start sections below.

Cluster requirements: - Privileged pods must be allowed (nvml-mock DaemonSet uses privileged: true for mknod) - containerd with NRI enabled, or nri.enabled=false; see Set up NRI injection - For DRA: Kubernetes 1.32+ with DynamicResourceAllocation feature gate enabled

Walkthroughs

Each of these installs the chart against a real consumer, end to end:

Guide	What it stands up
NVIDIA Device Plugin	Mock GPUs advertised as nvidia.com/gpu, and a workload scheduled against them
NVIDIA DRA Driver	Mock GPUs published as ResourceSlices, and a pod scheduled through a ResourceClaim
NVIDIA GPU Operator	The full operand stack — device plugin, GFD, DCGM and the validator

The rest of this page is the reference: what the chart installs, every value it takes, and the behaviours worth knowing before you change one.

Integration: fake-gpu-operator

fake-gpu-operator by Run:ai simulates GPUs at the Kubernetes API level for scale testing. nvml-mock can provide driver-level fidelity (real NVML API) on real nodes while fake-gpu-operator handles KWOK virtual nodes.

Enable Profile Discovery

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set integrations.fakeGpuOperator.enabled=true

This creates per-profile ConfigMaps in the shape fake-gpu-operator's loader reads:

kubectl -n mokka get cm -l fake-gpu-operator/gpu-profile=true

NAME                              DATA   AGE
gpu-profile-a100                  1      10s
gpu-profile-h100                  1      10s
gpu-profile-b200                  1      10s
gpu-profile-gb200                 1      10s
gpu-profile-gb300                 1      10s
gpu-profile-l40s                  1      10s
gpu-profile-t4                    1      10s

FGO loads these by name from its own namespace, so set integrations.fakeGpuOperator.targetNamespace to FGO's release namespace for them to be found. That requires FGO's builtinProfiles.enabled=false, because their builtin set uses the same seven names. See the fake-gpu-operator guide.

Custom Labels

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set integrations.fakeGpuOperator.enabled=true \
  --set 'integrations.fakeGpuOperator.profileLabels.my-org/gpu-profile=true'

InfiniBand mocking

Each profile carries an infiniband: block alongside the GPU config. When the DaemonSet starts, mock-ib reads it and writes a fake sysfs tree at /var/lib/nvml-mock/ib/sys/class/infiniband/.... Inside the container, three LD_PRELOAD shims cooperate (preload order libibmockumad.so:libibmockverbs.so:libibmocksys.so):

• libibmocksys.so rewrites every access to /sys/class/infiniband*, /sys/class/infiniband_mad/, /sys/class/infiniband_verbs/ and /dev/infiniband so sysfs-driven tools read from the rendered tree.
• libibmockumad.so proxies libibumad's umad_send / umad_recv to the in-pod mock-ib daemon (Unix socket) which handles SA path queries, ibping echoes, and SMP synthesis for iblinkinfo.
• libibmockverbs.so proxies open/read/write on /dev/infiniband/uverbsN so libibverbs consumers can enumerate HCAs.

POD=$(kubectl -n mokka get pods -l app.kubernetes.io/name=nvml-mock -o jsonpath='{.items[0].metadata.name}')

# sysfs / libibumad (always works):
kubectl -n mokka exec "$POD" -- ibstat
kubectl -n mokka exec "$POD" -- ibstatus

# libibverbs enumeration (modalias matches libmlx5's match table):
kubectl -n mokka exec "$POD" -- ibv_devinfo -l
kubectl -n mokka exec "$POD" -- ibv_devices

# Subnet management direct-route walk (cross-node fabric scan):
kubectl -n mokka exec "$POD" -- iblinkinfo

Full per-device ibv_devinfo (without -l) intentionally is not supported: after libibverbs claims the device, libmlx5's verbs_open_device issues real uverbs ioctl()s that a userspace LD_PRELOAD shim cannot fake. The same port-level information (state, phys state, GID, LID, rate, link layer) is available through ibstatus, which reads it from the rendered sysfs tree.

In NRI-injected pods

With nri.enabled=true the same tools are staged into the node overlay and reachable from any injected workload at /opt/nvml-mock/driver/usr/bin/<tool>. They carry their shared libraries (libibmad, libibumad, libibverbs, libnl) alongside them in driver/usr/lib64 and an RPATH of $ORIGIN/../lib64, so they run from an image that ships no InfiniBand stack of its own — a distroless or scratch workload, not just a full distro image.

Two limits apply there, both independent of the staging:

• ibstatus is a /bin/sh script rather than an ELF binary, so it needs an image with a shell.
• ibv_devinfo -l reports 0 HCAs found in an injected pod. Enumeration needs libibverbs to match the device to a provider driver (libmlx5), and the provider ships in the nvml-mock image rather than in the workload. Use ibstat -l, which reads the rendered sysfs through libibmocksys.so and lists every mock HCA.

The tools are glibc binaries. On a musl image (Alpine) they fail to exec at all, because PT_INTERP names /lib/ld-linux-*.so.* by absolute path and no RPATH can redirect that.

Defaults per profile

Profile	Enabled	HCA	Speed	HCAs per GPU
a100	yes	ConnectX-6 (MT4123)	HDR 200 Gb/s	1
h100	yes	ConnectX-7 (MT4129)	NDR 400 Gb/s	1
b200	yes	ConnectX-7 (MT4129)	NDR 400 Gb/s	1
gb200	yes	ConnectX-7 (MT4129)	NDR 400 Gb/s	1
gb300	yes	ConnectX-7 (MT4129)	NDR 400 Gb/s	1
l40s	no	—	—	—
t4	no	—	—	—

infiniband: block schema

Field	Default	Notes
enabled	false	Must be true to render any tree
hca_type	MT4129	Shows up as CA type in ibstat output
fw_version	28.39.2048	Firmware version
hw_rev	0x0	Hardware revision
board_id	MT_0000000838	Mellanox board ID
link_layer	InfiniBand	InfiniBand or Ethernet
rate_gbps	400	One of 100 (EDR), 200 (HDR), 400 (NDR), 800 (XDR)
port_state	ACTIVE	DOWN, INIT, ARMED, ACTIVE, ACTIVE_DEFER
phys_state	LinkUp	Disabled, Polling, Training, LinkUp, ...
hcas_per_gpu	1	Total HCAs = gpu.count * hcas_per_gpu
hca_count	0	If non-zero, used instead of gpu.count * hcas_per_gpu
guid_prefix	a088c20300ab	Hex prefix for node/port GUIDs. The renderer keeps the first 8 hex digits fixed and uses the lower 32 bits for node/HCA identity
node_desc_template	{node_name} mlx5_{idx}	{node_name} and {idx} are interpolated

NIC identity, locality and counters

Beyond what ibstat reads, the tree carries the files NIC health monitors and exporters use to discover a NIC, place it, and watch it:

• mlx5_N/device/ is the HCA's PCI function: vendor (0x15b3), device, numa_node, and a uevent carrying PCI_SLOT_NAME. Each HCA is paired with a GPU, takes the first free PCI bus after that GPU's, and reports the GPU's NUMA node. On the a100 profile, mlx5_0 sits at 0000:08:00.0 beside the GPU at 0000:07:00.0. Without a GPU address to pair with, numa_node is -1.
• mlx5_N/ports/1/counters/* and hw_counters/* hold the standard IB counters and the mlx5 transport counters (rnr_nak_retry_err, local_ack_timeout_err, roce_slow_restart, ...).
• With link_layer: Ethernet, each HCA also gets a RoCE netdev under sys/class/net/, named the way systemd names it (enp8s0np0 for 0000:08:00.0). Its operstate is up when port_state is ACTIVE and down otherwise, and it carries carrier_changes and statistics/*. mlx5_N/device/net/<netdev> and <netdev>/device/infiniband/mlx5_N link the two, as the kernel's device links do.

Every HCA is a physical function, so none has a device/physfn link. Counters read 0, and link_layer, port_state and phys_state apply to every HCA on the node.

Disable IB on a profile

Two options, depending on intent:

• Turn the in-pod mock IB off at runtime (no sysfs render, no mock-ib daemon, shims become no-ops) without editing the profile:

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set gpu.profile=h100 \
  --set infiniband.mockTier=off

This disables ibstat / ibping / iblinkinfo mocking in the pod. The chart still renders the IB Service and NetworkPolicy because those track the profile's infiniband.enabled, not the tier — use the next option to drop them too.

• Set infiniband.enabled: false in the profile via a custom profile file (preferred for full control). Note that gpu.customConfig replaces the entire profile config rather than merging, so an inline --set-string 'gpu.customConfig=infiniband: { enabled: false }' would throw away all the GPU settings — pass a complete config file instead:

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set-file gpu.customConfig=my-h100-no-ib.yaml

PCIe topology mocking

Each profile carries a pcie_topology: block describing the host's PCI root-complex layout. The node daemon's pcibus simulator reads it and writes a fake sysfs tree at /var/lib/nvml-mock/sys/... matching what real Linux kernels expose. Topology-aware consumers (NVIDIA DRA driver, device plugins computing NUMA hints) resolve "which PCIe root complex a GPU lives on" via a standard readlink() + path parse against the rendered tree:

$ readlink /var/lib/nvml-mock/sys/bus/pci/devices/0000:07:00.0
../../../devices/pci0000:00/0000:07:00.0

$ cat /var/lib/nvml-mock/sys/devices/pci0000:00/0000:07:00.0/numa_node
0

Defaults per profile

Profile	Root complexes	NUMA nodes	GPUs per root	NVSwitch bridges
a100	2 (pci0000:00, pci0000:80)	2 (dual EPYC)	4	6
h100	2 (pci0000:00, pci0000:80)	2 (dual socket)	4	4
b200	2 (pci0000:00, pci0000:80)	2 (dual socket)	4	0
gb200	2 (pci0000:00, pci0000:40)	2 (one per Grace CPU)	2	0
gb300	2 (pci0000:00, pci0000:40)	2 (one per Grace CPU)	2	0
l40s	2 (pci0000:00, pci0000:80)	2 (dual socket)	4	0
t4	1 (pci0000:00)	1	4	0

The bridges sit on the first root complex, which is where a baseboard's switches are attached on real hardware. See NVSwitches on the PCI bus.

pcie_topology: block schema

pcie_topology:
  root_complexes:
    - id: "pci0000:00"            # sysfs root-complex dir, format "pciDDDD:BB"
      numa_node: 0                 # numa_node value for every child device
      devices:
        - "0000:07:00.0"           # canonical 4-digit-domain BDF
        - "0000:0F:00.0"
    - id: "pci0000:80"
      numa_node: 1
      devices:
        - "0000:87:00.0"
        - "0000:90:00.0"

Nothing validates the block, so a typo is not reported anywhere — not through the daemon's /healthz. What the daemon renders is the block reconciled against the devices NVML reports, which silently absorbs most mistakes:

• A bus_id that is not an address in the kernel's DDDD:BB:DD.F form is dropped, and the device is left out of the tree. This includes the 8-digit domain NVML reports through nvmlPciInfo_t.busId; bus_id carries the 4-digit form, as busIdLegacy does. A value that is not an address is one no consumer can look up, and it would otherwise become a directory name.
• A BDF listed under a root complex that no entry in devices[] claims is dropped, along with any root complex it leaves empty. This is also how gpu.count works: capping the device list leaves the layout untouched, and the uncapped BDFs disappear from the tree rather than rendering as GPUs NVML denies exist.
• A BDF listed under two root complexes stays under the first and is dropped from the second.
• A device in devices[] whose BDF no root complex lists is still rendered, under the root its own address implies (pciDDDD:BB) and reporting numa_node -1. A GPU missing from the tree is one no consumer can resolve.

If a profile omits pcie_topology: entirely the renderer falls back to a flat single-root layout (every device under pci0000:00, NUMA 0). A profile whose devices declare no bus_id at all renders nothing, and the simulator empties any tree a previous profile left behind.

NVSwitches on the PCI bus

A GPU is not the only NVIDIA device on an HGX node. The baseboard's NVSwitches are PCIe endpoints too, and lspci lists them as bridges beside the GPUs:

$ lspci | grep NVIDIA
05:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch]
...
1a:00.0 3D controller: NVIDIA Corporation GH100 [H100 SXM5 80GB]

A switch reaches the tree by declaring a device_id in the profile's nvlink.switches list. List the same BDF under a pcie_topology: root complex to give the switch that root's numa_node; otherwise it lands under the root its address implies with numa_node -1, exactly as an unplaced GPU does.

The switch renders with PCI class 0x068000 (base class 0x06 bridge, subclass 0x80), while a GPU renders 0x030200 (3D controller). The distinction is not cosmetic: GPU Feature Discovery derives nvidia.com/gpu.mode from this class, so a switch enumerating as a 3D controller would read as an extra GPU.

Rack-scale platforms have no PCIe-visible switches

gb200 and gb300 declare NVSwitches without a device_id. On NVL72 the switches live in their own switch trays, reached over the NVLink cable cartridge, so a compute tray's lspci shows its GPUs and no bridges. Those switches still act as NVLink remote endpoints, which is what produces the NV18 all-to-all in nvidia-smi topo -m.

PCI sysfs in containers

Reaching the tree through MOCK_PCI_ROOT requires the libmockfs.so LD_PRELOAD shim, which only works for libc consumers: Go's os package issues openat directly, so the shim never sees the open and the process reads the node's real /sys, where the mock GPUs do not exist. GPU Feature Discovery and the NVIDIA DRA driver are both Go.

So the nvidia.com/gpu CDI spec the node daemon writes bind-mounts the tree read-only at the kernel paths:

Host	Container
/var/lib/nvml-mock/sys/devices	/sys/devices
/var/lib/nvml-mock/sys/bus/pci/devices	/sys/bus/pci/devices

Both, always together. The entries under sys/bus/pci/devices are relative symlinks into ../../../devices/pciDDDD:BB, so mounting that directory alone yields entries that list but whose every attribute read fails with ENOENT.

Trade-off: /sys/devices is mounted whole, which hides the host's other device classes — CPU topology among them — from served containers. Narrowing it to the profile's root complexes is not possible: the runtime would have to create the mountpoint, and sysfs is read-only in a container, so container creation fails outright rather than degrading. Profiles routinely declare root complexes the node does not have. #689 tracks removing the trade-off.

A workload that needs the node's real device tree must not request nvidia.com/gpu, since the mount rides the CDI spec the container toolkit resolves for that resource. nri.excludedNamespaces is not an escape: it only reaches the NRI plugin, whose own nvml-mock.nvidia.com/gpu spec carries the module tree but no PCI sysfs mounts.

Machine type (nvidia.com/gpu.machine)

GFD derives the label from --machine-type-file, which defaults to /sys/class/dmi/id/product_name — a path no mock can own under kind: the node image writes kind there and re-binds it into every container after the container's own mounts are set up, and on hosts without DMI (Docker Desktop) it does not exist at all.

The daemon therefore writes the machine type to driver/config/machine-type, and the CDI mount that carries config.yaml serves it at /etc/nvml-mock/machine-type. GFD has to be pointed at it by hand, with or without nri.enabled. The toolkit resolving nvidia.com/gpu applies the spec's mounts and drops its env (#747). The NRI plugin sets GFD_MACHINE_TYPE_FILE only in the containers it injects, which hold a GPU allocation or opt in by annotation (see Which containers are injected), and GFD does neither:

gfd:
  env:
    - name: GFD_MACHINE_TYPE_FILE
      value: "/etc/nvml-mock/machine-type"

The value is the profile's GPU product name, so gpu.machine matches gpu.product (NVIDIA-GB300-NVL) rather than the NVIDIA-GB300-NVL72 a real tray reports; the profiles carry no platform name to use instead.

Cross-node ibping

Sysfs mocking alone lets ibstat / iblinkinfo work, but real ibping needs UMAD I/O. For IB-enabled profiles, the chart preloads libibmockumad.so alongside libibmocksys.so, starts mock-ib in each pod, and exposes a headless Service on port 18515 for TCP fabric relay between nvml-mock pods.

The fabric listener binds 0.0.0.0 with no authentication, so the chart also ships a NetworkPolicy (infiniband.ping.networkPolicy.enabled, default true) that allows inbound fabric traffic only from peer nvml-mock pods. NetworkPolicy is enforced only by CNIs that implement it; Kind's default kindnet ignores it, so it is a no-op in the typical Kind fixture but limits exposure on Calico/Cilium-backed clusters. mock-ib is a test fixture — don't deploy it to a shared or production cluster.

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set gpu.profile=a100 \
  --set gpu.count=2 \
  --wait --timeout 120s

On a multi-node cluster, pick two nvml-mock pods on different nodes. Read the server LID from sysfs and ping that LID from the client:

SERVER_POD=$(kubectl -n mokka get pods -l app.kubernetes.io/name=nvml-mock \
  -o jsonpath='{.items[0].metadata.name}')
CLIENT_POD=$(kubectl -n mokka get pods -l app.kubernetes.io/name=nvml-mock \
  -o jsonpath='{.items[1].metadata.name}')

LID=$(kubectl -n mokka exec "$SERVER_POD" -- sh -c \
  "tr -d '[:space:]' < /var/lib/nvml-mock/ib/sys/class/infiniband/mlx5_0/ports/1/lid")

kubectl -n mokka exec "$CLIENT_POD" -- ibping -c 3 "$LID"

For automated cross-node validation (including peer restart and retries), run the ibping and ibfabric specs of the Go e2e suite. LID-based ping is the supported path; cross-node ibping -G <port_guid> is supported (use 0x hex without colons). The same specs cover the companion fabric tools:

• iblinkinfo and ibnetdiscover — the directed-route walk reaches peer GUIDs without duplicate-port errors.
• ibv_devinfo -l claims every rendered HCA via libmlx5, and ibstatus confirms ACTIVE / LinkUp port state.
• sminfo reports a master subnet manager, and every pod names the same one.

The mock reads MOCK_IB, MOCK_IB_PING_FABRIC, MOCK_IB_ROOT and MOCK_IB_PEERS. The chart derives the first three from the infiniband values.

Device injection mode

Applies only when nri.enabled=true, and only to the nri.deviceAnnotation opt-in path.

nri.deviceInjectionMode selects how the plugin delivers mock GPU device nodes to a container that carries nvml-mock.nvidia.com/devices: "true" and holds no GPU allocation:

Mode	Mechanism	Needs
raw (default)	The plugin stages the /dev/nvidia* nodes itself, in the NRI adjustment.	Nothing.
cdi	The plugin emits the CDI device nvml-mock.nvidia.com/gpu=all and the runtime resolves it from the spec the cdi simulator stages at <cdiSpecDir>/nvml-mock-nri.yaml.	A runtime with CDI on, and NRI that accepts CDI devices: not containerd 1.7.

Both modes deliver the same GPUs, and both inject libmockfs.so, so a libc reader finds the simulated modules either way. Only cdi bind-mounts /sys/module, so a Go reader finds them only there. cdi also sets NVML_MOCK_DEVICE_SOURCE=cdi, the only way to tell which mechanism ran.

CDI needs no container toolkit on the node. containerd 2.x enables CDI by default (enable_cdi = true, spec dirs /etc/cdi and /var/run/cdi), which includes the stock kindest/node image. containerd 1.7 cannot use cdi, even with enable_cdi: its NRI predates CDI device adjustments and silently drops the reference, so the container gets the mock library but no /dev/nvidia* nodes. raw stays the default.

If cdi is selected and no spec is staged, the plugin logs a warning and falls back to raw. It does not fail the pod: an unresolvable CDI device makes containerd reject container creation outright.

Neither mode changes which GPUs an allocated container gets. A container that holds a device-plugin or NVIDIA DRA allocation keeps exactly that allocation in both modes, even with the annotation, per MEP-0002.

NRI pod lifecycle

Applies only when nri.enabled=true.

The node agent and the NRI plugin run as separate containers in the same node DaemonSet pod. On Kubernetes 1.29 and later, the node agent is a restartable init container and NRI a regular container, which orders them:

• Startup. The node agent's startup probe reads /stagedz, which opens once its first Stage wave has written the driver tree, so Kubernetes starts NRI only after that. /stagedz does not wait for an intentional fabricmanager.initDelay, so new workloads can still observe the simulated IN_PROGRESS fabric state. Kubernetes runs no liveness probe until the startup probe passes, so nodeAgent.nriStartupTimeoutSeconds (120 by default) is also how long a node agent whose first staging failed waits before it is restarted.
• Termination. Kubernetes stops NRI before the node agent, so the plugin disconnects before the agent removes the staged driver tree.

While both run, NRI checks the node agent before each adjustment and leaves new containers unmodified while the agent is restarting or restaging; see Failing open.

If a cluster administrator has explicitly disabled the SidecarContainers feature gate, set nri.nativeSidecar=false. That setting, and Kubernetes 1.28, render both as regular containers, which Kubernetes does not order, so NRI leaves containers unmodified while the node agent stages files at startup. The chart picks the layout from Helm's .Capabilities.KubeVersion: pass --kube-version to helm template, or the equivalent setting in a GitOps renderer, when rendering offline. An upgrade across Kubernetes 1.29 switches the layout on the next Helm upgrade and rolls every node pod.

Changing an nri.* value rolls the node DaemonSet and briefly rebuilds the staged driver tree. The chart has no option to deploy the plugin separately, so this is the operational cost of running NRI.

Readiness is shared as well. A plugin that is not Ready, including one on a node whose container runtime has NRI disabled, marks the whole node pod NotReady; see NRI plugin failure modes.

For InfiniBand-enabled profiles, the headless -ibping Service publishes pod addresses even when the shared pod is NotReady. The relay runs in the node agent, so an unready NRI container must not hide an otherwise healthy relay from peer discovery. Kubernetes readiness is pod-wide, so this also publishes an address while the node agent itself is unready; relay clients already retry unreachable peers.

NRI plugin failure modes

Applies only when nri.enabled=true.

The NRI plugin injects the mock GPU stack at container-creation time, which is what lets a pod given GPUs the usual way see mock GPUs without a pod-spec change. It also means the injection is written into the container's OCI spec once, at creation. A pod that is already running keeps everything it was given, whatever happens to the plugin afterwards. Only pods created after a failure are affected, and they are affected silently.

That is the property that makes this worth hardening: a test suite that creates its pods early and asserts against them later keeps passing on a node where injection stopped hours ago.

The two modes

	Fail-closed	Fail-open
What the runtime does	Refuses to create the container	Creates the container without the plugin's adjustment
What you see	Pods stuck in ContainerCreating / CreateContainerError	Pods start normally
What the workload gets	Nothing — it never runs	A container with no mock GPU stack
Risk	Test runs stop	Test runs continue and report results that no longer mean what they claim

Fail-closed is loud and self-announcing. On a dedicated test cluster it is arguably the preferable posture: a broken mock stops the run instead of corrupting it.

Fail-open is the dangerous one, and it is the mode this chart is built to survive. It cannot be prevented from the plugin side — the decision belongs to containerd, not to the plugin — so the chart's posture is: assume fail-open can happen, and make it impossible for it to happen quietly.

Posture this chart targets

Detectable fail-open. Both probes exist to convert a silent window into a visible one, not to prevent it:

• Readiness (/readyz) reports serving only while the plugin is registered with the runtime and its handler is answering. Any window in which the node is not injecting shows up as a NotReady pod and a short DaemonSet count. Readiness restarts nothing; it is purely the detection surface.
• Liveness (/healthz) fails only when a container-creation request has been in flight past the wedge threshold, and restarts the container into a fresh registration.

Neither probe reduces to "is the process alive", because the process stays alive in every mode that matters:

State	Process	Connection	/readyz	/healthz
Registered and serving	up	up	200	200
Started, not yet registered	up	—	503	200
Unregistered by the runtime	up	dropped	503	200
Handler wedged	up	up	503	503 → restart

A wedged handler is the case that defeats every simpler check: pgrep nri-plugin finds the process, and a plain TCP check finds the socket bound, in exactly the state where nothing is being injected.

Losing the connection is deliberately not a liveness failure. The NRI stub's Run returns when the connection drops and the plugin exits on its own, so the kubelet already restarts it; failing liveness on "not registered" as well would only add restart loops whenever containerd is slow to come up.

containerd plugin_request_timeout

The wedge threshold is not a constant in the chart. The plugin derives it from the request timeout containerd itself reports at registration, and trips at twice that value. Past one whole timeout the runtime has already abandoned the request, so the container it belonged to was created without injection whatever happens next; the second is tolerance, so a single slow-but-completing request cannot restart the plugin.

NRI's defaults, from containerd/nri/pkg/api:

Setting	Default
plugin_request_timeout	2s (wedge threshold 4s)
plugin_registration_timeout	5s

Both are set on the runtime, not in this chart:

[plugins."io.containerd.nri.v1.nri"]
  disable = false
  socket_path = "/var/run/nri/nri.sock"
  # Raise only if the plugin legitimately needs longer than 2s to answer.
  plugin_request_timeout = "2s"

Guidance:

• Leave it at the default unless you have evidence. The plugin's CreateContainer path does a stat of the topology document, a directory read of the device directory, and one stat per device node — all against the hostPath-mounted overlay, and nothing else. On a healthy node that is well under 2s. A raised timeout does not make injection more reliable; it widens the window in which each container creation blocks on a plugin that may already be wedged.
• Those filesystem calls are the realistic wedge. They are the only blocking operations in the handler, so an overlay backed by a hung mount is how this plugin stops answering while staying alive and connected.
• Raising it widens the wedge threshold automatically. No chart change is needed, and none should be made — nri.livenessProbe tuning and plugin_request_timeout are not independent knobs.
• Do not raise it to paper over a wedge. A plugin that needs more than 2s is the failure this hardening detects, not a tuning problem.

Checking a node by hand

# Which nodes are actually injecting right now
kubectl get pods -n mokka -l app.kubernetes.io/name=nvml-mock -o wide

# Why a given node is not
kubectl describe pod -n mokka <nvml-mock-pod>

Both probe endpoints answer with the reason in the body, so a readiness failure in kubectl describe reads as not registered with the container runtime; new containers are not being injected rather than a bare status code.

The port is not reachable from the node: the node DaemonSet does not set hostNetwork, so nri.healthPort is bound only inside the pod's own network namespace, on the pod IP where the kubelet reaches it.

Configuration

Values

Parameter	Default	Description
gpu.profile	gb300	GPU profile: a100, h100, b200, gb200, gb300, l40s, or t4
gpu.count	""	Number of mock GPUs per node. Empty derives it from the profile's devices: list (8 for the baseboard profiles, 4 for t4, gb200 and gb300); a larger value is capped to that list at runtime
gpu.customConfig	""	Inline YAML to override profile config entirely
gpu.dynamicMetrics.enabled	false	Make the mock return time-varying temperature / power / utilization readings instead of the static profile values. See Dynamic Metrics below.
gpu.dynamicMetrics.seed	0 (baseline)	RNG seed; 0 uses a time-based seed, non-zero produces reproducible sequences.
gpu.dynamicMetrics.temperature.*	baseline (base_c: 55, …)	base_c, variance_c, ramp_c, ramp_period_sec for the GPU temperature generator.
gpu.dynamicMetrics.power.*	profile default, else baseline 250000/25000	base_mw, variance_mw for the power generator (clamped to the profile's min/max_limit_mw). Resolved baseline < profile default < user override; profiles outside the 250W baseline set their own (t4 ~65W, b200/gb200 ~600W, gb300 ~800W). See Dynamic Metrics.
gpu.dynamicMetrics.utilization.*	baseline (pattern: burst, …)	pattern (idle | busy | burst | steady), gpu_min/max, memory_min/max, burst_period_sec.
gpu.failureInjection.enabled	false	Enable simulated GPU failures (lost / fallen off bus / uncorrectable ECC). See Failure Injection below.
gpu.failureInjection.mode	healthy	Failure mode: healthy (default, no-op), lost, fallen_off_bus, or ecc_uncorrectable. With the inert default, enabled: true alone produces a healthy device — you must set mode explicitly to engage failures.
gpu.failureInjection.probability	0.0	Per-call probability [0, 1] for stochastic failure activation.
gpu.failureInjection.after_calls	0	Activate failure deterministically after N guarded NVML calls (0 = disabled).
gpu.failureInjection.seed	0	RNG seed for probability rolls; 0 uses a time-based seed.
gpu.failureInjection.xid.code	0	Xid error code delivered via the NVML event set (NVML_EVENT_TYPE_XID_CRITICAL_ERROR) once tripped. 0 = no Xid.
global.imageRegistry	""	Registry every chart image is pulled from, such as a mirror. Replaces the registry host of image.repository, nri.image.repository and controlPlane.image.repository, so ghcr.io/nvidia/nvml-mock becomes <registry>/nvidia/nvml-mock.
global.imagePullSecrets	[]	Pull secrets added to every chart pod (nvml-mock, NRI and control plane), ahead of imagePullSecrets. Each entry is a Secret name or {name: <secret>}.
imagePullSecrets	[]	Pull secrets added to every chart pod, after global.imagePullSecrets; duplicates are dropped. Same entry forms.
extraObjects	[]	Additional Kubernetes objects rendered with the release, each passed through tpl so it can use chart values such as {{ .Release.Name }}. A list, or a map keyed by name so layered values files can override an entry or drop it with "". Each entry is an object or a YAML string.
image.repository	ghcr.io/nvidia/nvml-mock	Container image repository
image.digest	""	Immutable sha256:... digest. When set, pins the image and takes precedence over image.tag.
image.tag	"" (chart appVersion)	Container image tag. When empty, the chart appVersion: the release version in a released chart, the next -dev version on main.
image.pullPolicy	IfNotPresent	Image pull policy. When empty, the Kubernetes default for the rendered image: Always for the latest tag, IfNotPresent otherwise.
featureGates	{}	Feature gates passed to the node agent, NRI plugin and control plane, as Name: true\|false. See Feature Gates.
driverVersion	"" (auto)	NVIDIA driver version to mock. When empty, read from system.driver_version of the resolved GPU config (the selected gpu.profile file, or gpu.customConfig if set), so the profile is the single source of truth (e.g. GB200 → 580.65.06, B200 → 560.35.03, GB300 → 580.65.06, others → 550.163.01). Set explicitly only to override the profile.
nodeSelector	{}	Node selector for DaemonSet
tolerations	[{operator: Exists}]	Pod tolerations (default: tolerate all)
priorityClassName	""	PriorityClass for the node DaemonSet pods. The node agent stands in for its node's GPU driver, so system-node-critical keeps it from being starved or evicted before the workloads that depend on it.
affinity	{}	Pod affinity for the node DaemonSet pods
podAnnotations	{}	Annotations added to the node DaemonSet pods. The chart's own annotations (checksum/config, checksum/mig-profiles, kubectl.kubernetes.io/default-container) always win.
podLabels	{}	Labels added to the node DaemonSet pods. Must not set a selector label (app.kubernetes.io/name, instance or component).
nodeAgent.livenessProbe	httpGet /healthz on health	Node agent liveness probe. Set to null to drop it.
nodeAgent.readinessProbe	httpGet /readyz on health	Node agent readiness probe. Set to null to drop it.
nodeLabels.featuresDir	/etc/kubernetes/node-feature-discovery/features.d	Host directory NFD's local source reads feature files from. Override only if NFD runs with a non-default featureFilesDir
nodeAgent.nriStartupTimeoutSeconds	120	On Kubernetes 1.29+, maximum time the node-agent /stagedz gate may wait for its first staging and apply cycle before Kubernetes restarts it when NRI uses native sidecars
integrations.fakeGpuOperator.enabled	false	Create per-profile ConfigMaps named gpu-profile-<profile>, keyed profile.yaml, in the shape fake-gpu-operator's loader reads
integrations.fakeGpuOperator.targetNamespace	"" (release namespace)	Namespace for the profile ConfigMaps. Set to FGO's release namespace for FGO to find them; requires FGO's builtinProfiles.enabled=false to avoid a Helm ownership collision on the same seven names
integrations.fakeGpuOperator.profileLabels	{"run.ai/gpu-profile": "true"}	Extra labels on profile ConfigMaps. The contract labels fake-gpu-operator/gpu-profile and nvml-mock/profile-name are always emitted and cannot be removed here
infiniband.mockTier	"" (auto)	MOCK_IB tier: off, sysfs, or full. Empty auto-derives full for IB-enabled profiles and sysfs otherwise (keeps the libibmocksys redirect active so any real host IB is masked). off makes every shim a no-op and skips the daemon. An invalid value fails helm template
infiniband.ping.port	18515	TCP port for fabric relay between nvml-mock pods (mock-ib / ibping always enabled)
infiniband.ping.networkPolicy.enabled	true	Restrict inbound access to the fabric port to peer nvml-mock pods. No-op on CNIs that don't enforce NetworkPolicy (e.g. Kind's kindnet)
nri.enabled	true	Run the nvml-mock-nri containerd NRI plugin in the node DaemonSet. In non-excluded namespaces it injects containers that hold a GPU allocation or whose pod opts in by annotation; see Set up NRI injection. Set false where containerd has NRI disabled. Always install into a dedicated namespace (-n mokka) to avoid excluding default
nri.nativeSidecar	true	On Kubernetes 1.29+, use the ordered SidecarContainers layout. Set to false when that feature gate is explicitly disabled; the chart falls back to unordered regular containers
nri.socketPath	/var/run/nri/nri.sock	NRI socket on the host. Its directory is hostPath-mounted into the plugin
nri.pluginName / nri.pluginIndex	nvml-mock / "10"	NRI registration identity. The index orders this plugin against others
nri.overlay.hostPath / nri.overlay.mountPath	/var/lib/nvml-mock / /opt/nvml-mock	Host overlay staged by the main DaemonSet, and the path it is injected at inside workloads
nri.optOutAnnotation	nvml-mock.nvidia.com/inject	Pod annotation; value false disables injection for that pod
nri.deviceAnnotation	nvml-mock.nvidia.com/devices	Pod annotation; value true gives a pod with no GPU allocation the overlay and every mock GPU on the node. Ignored for containers that hold an allocation. Pod-authored, so treat it as part of the demo trust boundary
nri.deviceInjectionMode	raw	How nri.deviceAnnotation delivers GPUs: raw stages the device nodes directly, cdi emits a CDI device reference the runtime resolves. See Device injection mode
nri.cdiSpecDir	/var/run/cdi	Host directory holding CDI specs, mounted read-only into the plugin. Must be one of the runtime's configured cdi_spec_dirs
nri.imexChannelAnnotation	nvml-mock.nvidia.com/imex-channels	Pod annotation; value true gives the pod the mock /dev/nvidia-caps-imex-channels/channelN nodes staged by imex.mockChannels (no channels when that is disabled), and nothing else. Same trust boundary as nri.deviceAnnotation
nri.infinibandAnnotation	nvml-mock.nvidia.com/infiniband	Pod annotation; value true gives the pod the overlay with the mock InfiniBand tools active (MOCK_IB=full). Without mock GPUs unless the pod also selects them. Same trust boundary as nri.deviceAnnotation
nri.excludedNamespaces	[]	Extra namespaces to skip. The release namespace and kube-system are always excluded
nri.healthPort	8080	Port serving /healthz and /readyz. Bound only in the pod's network namespace — this DaemonSet does not use hostNetwork, so nothing is exposed on the node
nri.readinessProbe	/readyz, periodSeconds: 10, failureThreshold: 2	Detects that the node has stopped injecting. Set to null to drop. See NRI plugin failure modes
nri.livenessProbe	/healthz, periodSeconds: 10, failureThreshold: 3	Restarts a wedged plugin. Threshold follows containerd's plugin_request_timeout; do not tune the two independently. Set to null to drop
nri.resources	requests cpu: 10m, memory: 32Mi	Resource requests/limits for the plugin container. No limits by default

Node Labels

nvml-mock writes no node labels itself. It writes a feature file NFD turns into one label, and NFD retires that label on its next cycle once the daemon deletes the file at shutdown:

Label	Written by	Removed by
feature.node.kubernetes.io/pci-10de.present=true	NFD, from a feature file the node daemon writes	NFD, once the node daemon deletes the file

Labels under nvidia.com/ — gpu.present, gpu.count, gpu.product — belong to NFD and GFD exactly as on real hardware, and are absent unless those are deployed. Workloads that need to land on a mock node select the GPU node-pool label their cluster gives those nodes, as they would on real hardware.

The label above is produced by Node Feature Discovery. nvml-mock only supplies the input: internal/agent/pcibus writes pci-10de.present=true into nodeLabels.featuresDir, which NFD's local source reads and turns into the namespaced label. With no NFD on the cluster the file is inert and the label does not exist — which is the honest state, and is what the e2e in tests/e2e/go/scenario_nfd_test.go asserts.

NFD's PCI source still cannot see mock GPUs as deployed: it reads a host-prefixed sysfs path fixed at link time (/host-sys/bus/pci/devices, from HOSTMOUNT_PREFIX), while nvml-mock's rendered PCI tree lives under /var/lib/nvml-mock/sys and is reachable at the canonical /sys path only through an LD_PRELOAD sysfs shim — and nfd-worker ships statically linked, so LD_PRELOAD is inert in it. Either fact alone is enough; both hold (verified against NFD v0.19.0, the version pinned in go.mod).

That is a limit of how NFD is deployed, not of the rendered tree. Pointed at /var/lib/nvml-mock/sys, NFD v0.19.0 enumerates every mock GPU and — with the sources.pci.deviceLabelFields: [vendor] that GPU Operator configures — derives exactly pci-10de.present on its own. The rendered devices already carry all five attributes its PCI source treats as mandatory. Only visibility is missing, and nothing nvml-mock can do supplies it without editing a third party's DaemonSet, which is why the local source is the route the chart uses.

Writing a feature file rather than the label is why the DaemonSet needs no cluster RBAC at all: nothing in the pod calls the Kubernetes API.

GPU Profiles

Built-in profiles provide realistic hardware specs for common data center GPUs. Select a profile with --set gpu.profile=<name>:

# Deploy as an 8-GPU H100 node
helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set image.repository=nvml-mock \
  --set image.tag=local \
  --set gpu.profile=h100

# Deploy as a 4-GPU B200 node
helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set image.repository=nvml-mock \
  --set image.tag=local \
  --set gpu.profile=b200 \
  --set gpu.count=4

Profile Comparison

	A100	H100	B200	GB200	GB300	L40S	T4
Profile name	a100	h100	b200	gb200	gb300	l40s	t4
Full name	A100-SXM4-40GB	H100 80GB HBM3	B200	GB200 NVL	GB300 NVL	L40S	Tesla T4
Architecture	Ampere	Hopper	Blackwell	Blackwell	Blackwell Ultra	Ada Lovelace	Turing
Compute capability	8.0	9.0	10.0	10.0	10.0	8.9	7.5
CUDA cores	6,912	16,896	18,432	18,432	21,632	18,176	2,560
Memory	40 GiB HBM2e	80 GiB HBM3	180 GiB HBM3e	186 GiB HBM3e	278 GiB HBM3e	48 GiB GDDR6	16 GiB GDDR6
NVLink	v3, 12 links	v4, 18 links	v5, 18 links	v5, 18 links	v5, 18 links	—	—
NVLink BW	600 GB/s	900 GB/s	1.8 TB/s	1.8 TB/s	1.8 TB/s	—	—
TDP	400W	700W	1,000W	1,000W	1,400W	350W	70W
PCIe	Gen4	Gen5	Gen6	Gen6	Gen6	Gen4	Gen3
MIG instances	7	7	7	7	7	0	0
Grace CPU	—	—	—	Yes (NVLink-C2C)	Yes (NVLink-C2C)	—	—
FP8	—	Yes	Yes	Yes	Yes	Yes	—
FP4	—	—	Yes	Yes	Yes	—	—
FP6	—	—	—	—	Yes	—	—
Driver version	550.163.01	550.163.01	560.35.03	580.65.06	580.65.06	550.163.01	550.163.01

When to Use Each Profile

• a100 — broadest compatibility. Most NVIDIA software assumes A100 in docs and examples. Use it when a test targets Ampere or trips over newer architectures.
• h100 — testing Hopper-specific features: FP8, Transformer Engine, PCIe Gen5, or NVLink v4 topology.
• b200 — testing next-gen Blackwell features: FP4, NVLink v5, PCIe Gen6. Standalone GPU (no Grace CPU).
• gb200 — testing Grace-Blackwell Superchip: NVLink-C2C to Grace CPU, unified memory, and Blackwell features.
• gb300 (default) — testing Grace-Blackwell Ultra Superchip: 278 GiB HBM3e per GPU, 1.4 kW TDP, FP6 in addition to FP4/FP8, and Blackwell Ultra driver line (580.65.06).
• l40s — testing Ada Lovelace inference workloads: FP8, PCIe Gen4, no NVLink (PCIe-only topology).
• t4 — testing Turing inference GPUs: low power (70W), small memory (16 GiB), 4 GPUs per node.

Custom Configuration

For GPU types not covered by built-in profiles, provide your own config YAML.

Option A: File-based (recommended)

Create a YAML file following the profile format, then pass it at install time:

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set image.repository=nvml-mock \
  --set image.tag=local \
  --set-file gpu.customConfig=my-custom-gpus.yaml

Option B: Inline values

For small overrides, embed the config directly in a values file:

# custom-values.yaml
gpu:
  count: 4
  customConfig: |
    version: "1.0"
    system:
      driver_version: "550.163.01"
      nvml_version: "12.550.163.01"
      cuda_version: "12.4"
      cuda_version_major: 12
      cuda_version_minor: 4
    device_defaults:
      name: "NVIDIA L40S"
      architecture: "ada_lovelace"
      compute_capability:
        major: 8
        minor: 9
      num_gpu_cores: 18176
      memory:
        total_bytes: 48318382080
        reserved_bytes: 536870912
        free_bytes: 47781511168
        used_bytes: 0
    devices:
      - index: 0
        uuid: "GPU-14050000-0000-0000-0000-000000000000"
        minor_number: 0

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set image.repository=nvml-mock \
  --set image.tag=local \
  -f custom-values.yaml

Writing a Custom Profile

Use an existing profile as your starting point:

cp deployments/nvml-mock/helm/nvml-mock/profiles/a100.yaml my-custom-gpus.yaml

Key fields to change:

Field	What to set
device_defaults.name	GPU name shown in nvidia-smi
device_defaults.architecture	Architecture string (ampere, hopper, blackwell, etc.)
device_defaults.compute_capability	major / minor version
device_defaults.num_gpu_cores	CUDA core count
device_defaults.memory.total_bytes	Total GPU memory in bytes
devices	One entry per GPU (match gpu.count) with unique UUIDs
nvlink	NVLink version and links (or omit for PCIe-only GPUs)

The full YAML schema matches the fields exposed by nvidia-smi -x -q. See the built-in profiles in deployments/nvml-mock/helm/nvml-mock/profiles/ for complete examples.

Dynamic Metrics

Real GPUs report metrics that change over time — temperature rises under load, utilization fluctuates, power draw ramps. By default the mock is fully static: whatever values are set in a profile's thermal, power, and utilization sections are returned unchanged on every call.

Set gpu.dynamicMetrics.enabled=true to have the rendered ConfigMap inject a device_defaults.dynamic_metrics block. The mock then returns fluctuating values from GetTemperature, GetPowerUsage, and GetUtilizationRates. Each sub-section (temperature, power, utilization) can be tuned independently; the overlay works with any built-in profile and with gpu.customConfig.

Each field resolves in three layers, highest priority last:

chart baseline  <  GPU profile default  <  user override (values / --set)

The profile layer matters for power: one global base_mw can't fit every profile's [min_limit_mw, max_limit_mw] envelope, so profiles outside the 250W baseline declare their own base via a Helm-only dynamic_metrics_defaults key (t4 ~65W, b200/gb200 ~600W, gb300 ~800W). The engine ignores that key; it only takes effect once enabled: true folds it into dynamic_metrics.

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set image.repository=nvml-mock \
  --set image.tag=local \
  --set gpu.profile=h100 \
  --set gpu.dynamicMetrics.enabled=true \
  --set gpu.dynamicMetrics.utilization.pattern=burst

Or via a values file:

gpu:
  profile: h100
  dynamicMetrics:
    enabled: true
    seed: 0                             # set non-zero for reproducibility
    temperature:
      base_c: 60
      variance_c: 3
      ramp_c: 15
      ramp_period_sec: 120
    power:
      base_mw: 500000
      variance_mw: 50000
    utilization:
      pattern: burst                    # idle | busy | burst | steady
      gpu_min: 0
      gpu_max: 100
      memory_min: 0
      memory_max: 100
      burst_period_sec: 30

Utilization pattern semantics (values are always clamped to 0..100):

pattern	sampled from
idle	bottom quarter of [gpu_min, gpu_max]
busy	top quarter of [gpu_min, gpu_max]
burst	alternates idle / busy every burst_period_sec
steady	full [gpu_min, gpu_max] range (default if omitted)

See pkg/gpu/mocknvml/README.md for the full engine-side reference.

Failure Injection

Real GPUs occasionally fall off the bus, accumulate uncorrectable ECC errors, or surface Xid events. By default the mock reports healthy hardware. Set gpu.failureInjection.enabled=true to have the rendered ConfigMap inject a device_defaults.failure block; the mock will then trip the device into the configured failure mode based on the trigger you choose:

# Deterministic: device goes "lost" after the 200th NVML call
helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --namespace mokka --create-namespace \
  --set gpu.profile=h100 \
  --set gpu.failureInjection.enabled=true \
  --set gpu.failureInjection.mode=lost \
  --set gpu.failureInjection.after_calls=200

# Stochastic + Xid: 1% chance per call to surface ECC double-bit (Xid 64),
# bounded to trip within 10k calls so CI does not hang.
gpu:
  profile: h100
  failureInjection:
    enabled: true
    mode: ecc_uncorrectable
    probability: 0.01
    after_calls: 10000
    seed: 12345
    xid:
      code: 64

Per-mode behaviour:

mode	guarded API calls return	handle lookup returns	identity getters	ECC counters	event set
healthy (default)	normal values	normal handle	normal values	zero	empty
lost	ERROR_GPU_IS_LOST	ERROR_GPU_IS_LOST	ERROR_GPU_IS_LOST	error	ERROR_GPU_IS_LOST (Xid once first if xid: set and getter-tripped)
fallen_off_bus	ERROR_GPU_IS_LOST	ERROR_GPU_IS_LOST	ERROR_GPU_IS_LOST	error	ERROR_GPU_IS_LOST (Xid once first if xid: set and getter-tripped)
ecc_uncorrectable	normal values	normal handle	normal values	strictly-increasing	one XID_CRITICAL_ERROR if xid

A configured Xid is delivered once per trip, through either nvmlEventSetWait_v1 or nvmlEventSetWait_v2. Subsequent waits then match the rest of the mode: lost and fallen_off_bus return NVML_ERROR_GPU_IS_LOST immediately, as real NVML does after Xid 79; ecc_uncorrectable reports NVML_ERROR_TIMEOUT (no event). With no event pending and no lost device the wait blocks for the caller's timeout, like real NVML — clients such as the device-plugin health monitor and dcgm-exporter loop on the wait with no sleep of their own, so an immediate TIMEOUT would spin a CPU core. ERROR_GPU_IS_LOST is the exception: real NVML returns it promptly, and those same clients already back off on it.

A device the config declares immediately lost — a bare mode: lost / fallen_off_bus block with no after_calls or probability gate — fails the wait with NVML_ERROR_GPU_IS_LOST on its own, so a client that only loops on nvmlEventSetWait (the DRA driver's health monitor) sees a lost GPU the way real NVML surfaces one that fell off the bus, without calling a getter first. Everything else still needs a trip: the injector trips on a guarded device call (GetTemperature, GetEccErrors, …), never on the wait itself, so delivering the configured Xid, the ecc_uncorrectable event, and any after_calls / probability gate only advance once something drives a device getter (nvidia-smi -q, a dcgm-exporter scrape). nvml-mock-ctl only writes the override file — it configures the failure, it does not trip it.

Values rendered into the ConfigMap are validated against values.schema.json at install / upgrade time: typos like mode: healhty or out-of-range values like probability: 1.5 are rejected by Helm before the chart renders, so misconfigurations surface as actionable schema errors instead of silent runtime coercion.

Failure injection composes with gpu.dynamicMetrics: with both enabled the device returns dynamic readings while healthy and switches to the configured failure mode once the trigger fires. Once tripped a device stays tripped for the lifetime of the pod, matching real hardware that needs a reboot to recover.

Verifying with nvidia-smi

Each nvidia-smi invocation is a fresh process whose call counter starts at 0, so a narrow query like --query-gpu=ecc.errors.uncorrected.aggregate.total will only ever issue one guarded call per GPU per invocation. (Within a single process the counter, like the rest of the device state, also survives nvmlShutdown() followed by nvmlInit().) To see the failure surface from a single short command set after_calls: 1, or use a richer query that issues several guarded calls per GPU (e.g. nvidia-smi -q) so the trigger fires within one process.

# mode: lost / fallen_off_bus  ─  handle lookup itself fails once tripped.
# nvidia-smi prints "Unable to determine the device handle for GPU ..."
# and exits non-zero.
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi -L
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi --query-gpu=name,uuid --format=csv
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi -q                # "GPU is lost"

# mode: ecc_uncorrectable  ─  device stays addressable; counters grow and
# nvmlEventSetWait_v1/_v2 delivers the configured Xid once per trip.
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi -q -d ECC
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi \
  --query-gpu=ecc.errors.uncorrected.aggregate.total --format=csv
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi \
  --query-gpu=ecc.errors.uncorrected.aggregate.dram  --format=csv

# Any mode  ─  watch the engine trip in real time.
kubectl -n mokka exec ds/nvml-mock -- env MOCK_NVML_DEBUG=1 \
  nvidia-smi -q -d ECC 2>&1 | grep -E 'failure|GPU_IS_LOST|Xid'

# One long-running process so the per-process call counter accumulates
# (useful when after_calls > 1 and you want to see a deterministic trip
# without restarting the daemonset).
kubectl -n mokka exec ds/nvml-mock -- nvidia-smi \
  --query-gpu=ecc.errors.uncorrected.aggregate.total --format=csv -l 1

See pkg/gpu/mocknvml/README.md for the full engine-side reference, including how the modes interact with specific NVML calls.

How It Works

The chart deploys:

1. DaemonSet — runs a privileged container on each node that:
2. Copies libnvidia-ml.so.{version} to the host at /var/lib/nvml-mock/driver/usr/lib64/
3. Creates symlinks (libnvidia-ml.so.1 → libnvidia-ml.so.{version})
4. Creates mock device nodes at /var/lib/nvml-mock/driver/dev/nvidia{N,ctl,-uvm,-uvm-tools} (CDI bind-mounts them to /dev/nvidia* in consumer containers)
5. Writes GPU config YAML at /var/lib/nvml-mock/driver/config/config.yaml
6. Writes the NFD feature file that makes feature.node.kubernetes.io/pci-10de.present=true appear — see Node Labels
7. ConfigMap — GPU configuration from the selected profile
8. ConfigMap (<fullname>-mig-profiles) — the selected board's MIG partition table, mounted at /etc/nvml-mock/mig and pointed at by MOCK_MIG_PROFILES_CONFIG. Rendered only for the five MIG-capable profiles, and never with gpu.customConfig, whose table is the user's — see where the table lives
9. ServiceAccount — no cluster RBAC; nothing in the pod calls the API

Consumer components (DRA driver, device plugin) mount /var/lib/nvml-mock and use --nvidia-driver-root=/var/lib/nvml-mock/driver to discover GPUs through standard NVML tryResolveLibrary paths.

Known Limitations

The mock NVML library covers the NVML C API surface used by consumers for GPU discovery and monitoring. Some host-level subsystems are not mocked:

What's Missing	Affected Consumer	Impact
/sys/bus/pci/devices/{busID} sysfs entries in a container the mock does not serve	Any Go consumer	Go's os package issues raw syscalls that the LD_PRELOAD shim cannot intercept, so a Go binary reads the host's real sysfs. Containers served the nvidia.com/gpu CDI spec get the tree bind-mounted at the kernel paths instead; ones reached only by the NRI plugin still see the host's — see PCI sysfs in containers
The host's other device classes in a container the mock does serve	Anything reading /sys/devices	/sys/devices is replaced wholesale, so CPU topology and the container's namespaced sysfs are not visible there. Tracked in #689
/sys/bus/pci/devices/{busID}/numa_node in a container the mock does not serve	Device plugin	NUMA-aware topology hints unavailable there; scheduling works but NUMA affinity not enforced. A served device plugin does get the hints — the renderer writes numa_node for every device
/sys/bus/pci/devices/*/vendor,device,class as NFD reads them (/host-sys/…, fixed at link time)	NFD (Node Feature Discovery)	PCI feature labels not auto-detected. pci-10de.present is created by NFD from a feature file nvml-mock drops in nodeLabels.featuresDir — see Node Labels

PCIe Root Complex (DRA driver)

When using the DRA driver with nvml-mock, you will see warnings like:

W0319 11:41:21.314205       1 nvlib.go:491] error getting PCIe root for device 0,
  continuing without attribute: failed to resolve PCIe Root Complex for PCI Bus ID
  0000:07:00.0: failed to read symlink for PCI Bus ID /sys/bus/pci/devices/0000:07:00.0:
  readlink /sys/bus/pci/devices/0000:07:00.0: no such file or directory

The driver resolves PCIe root complex topology by readlink()-ing /sys/bus/pci/devices/{busID}, and it is a Go binary, so libmockfs.so cannot redirect that read to the rendered tree. A container served the nvidia.com/gpu CDI spec gets the tree at that path and resolves the root; one the mock does not serve reads the node's real sysfs and logs the warning above. GPUs are fully allocatable either way — only the dra.k8s.io/pcieRoot attribute is affected.

What its absence blocks: DRA topology-aware scheduling that uses pcieRoot to align devices on the same PCIe root complex — for example, co-scheduling a GPU with an SR-IOV virtual function (VF) from the same root for optimal data path locality. Without pcieRoot, ResourceClaims that express cross-device topology constraints cannot be validated.

Confirming the attribute end-to-end against the pinned driver version is tracked in #265. Note the upstream constraint the shape of the tree has to satisfy: deviceattribute rejects a symlink whose resolved target does not start with devices/pci, so the mock's entries have to stay canonical relative symlinks and be served at the kernel path rather than redirected elsewhere.

Troubleshooting

ImagePullBackOff: Verify the image is accessible. By default the chart pulls ghcr.io/nvidia/nvml-mock:<chart appVersion>; check that tag exists or set image.tag. For local builds, ensure the image is loaded into your cluster (see Quick Start).

DaemonSet not ready: Check pod logs: kubectl -n mokka logs -l app.kubernetes.io/name=nvml-mock

GPU Operator operands stuck on toolkit-validation: Six operand DaemonSets block until /run/nvidia/validations/toolkit-ready exists. nvml-mock deliberately does not write it — the marker belongs to GPU Operator's own nvidia-validator, which deletes it, runs nvidia-smi against the mock driver, and recreates it only on success. Writing it here would let operands clear the gate before any check had run. If operands block on a marker that never returns while the validator is in CrashLoopBackOff, check CLEANUP_ALL: it makes the validator RemoveAll its output dir, which fails EBUSY on the bind mount and exits before recreating anything. Nothing sets it by default.

Device plugin shows 0 GPUs: Verify mock files exist on the node:

NODE_CONTAINER=$(docker ps --filter name=control-plane -q)
docker exec "$NODE_CONTAINER" ls /var/lib/nvml-mock/driver/usr/lib64/libnvidia-ml.so.*
docker exec "$NODE_CONTAINER" cat /var/lib/nvml-mock/driver/config/config.yaml

DRA driver pods not ready: Check DRA logs:

kubectl -n nvidia logs -l app.kubernetes.io/name=dra-driver-nvidia-gpu --tail=100

PCIe root warnings from DRA driver: See Known Limitations.

Privileged pods blocked: Your cluster may have PodSecurity or OPA/Gatekeeper policies blocking privileged: true. KIND allows this by default. For managed clusters, you may need to create a PodSecurity exception for the nvml-mock release namespace.

Related Documentation

• Mock NVML Library Documentation
• E2E Test Workflow
• KIND DRA Config
• Device Plugin Mock Manifest