Runtime control with `nvml-mock-ctl`¶

nvml-mock-ctl mutates the simulated GPU state of a running nvml-mock node without a Helm upgrade, image rebuild, or pod restart. Use it to inject failures, flip ECC state, or tweak metrics (temperature, power, utilization, clocks, fan, …) on the fly while a test is in flight.

Boot-time state comes from the Helm profile / config.yaml (see the Helm chart README and Configuration Reference).
Runtime state is layered on top by nvml-mock-ctl, described here.

How it works¶

nvml-mock is a per-process shared library (libnvidia-ml.so), not a daemon. There is no server to send commands to. Instead:

nvml-mock-ctl atomically writes a node-local config override file, overrides.yaml, that sits next to the pristine config.yaml.
The mock engine loaded inside every consumer process re-reads that config override on a short TTL (default 1s, MOCK_NVML_OVERRIDES_TTL) and deep-merges it over the pristine base config. The base config.yaml is never mutated.
Because the config override is bind-mounted into consumer containers (via CDI), both already-running processes and freshly-started ones converge on the new state within one TTL.

The merge order is:

base config.yaml  <  config override "all:" block  <  config override "devices[<idx>]:" block

So a per-device override wins over an all override, which wins over the base profile. Unknown fields and bad types are rejected at write time, so a typo fails the command instead of silently doing nothing.

v1 scope — what is and isn't hot-reloadable¶

The config override drives every config-derived getter: failure injection, ECC mode/counters, temperature, power, utilization, clocks, fan, performance state, device memory (memory.total_bytes / free_bytes / used_bytes / reserved_bytes / memory_bus_width), and the like. These change within one TTL.

Memory values are reported verbatim from the effective config — setting memory.used_bytes alone does not recompute memory.free_bytes. Set both in one command to keep the reported triple coherent:

nvml-mock-ctl set --gpu 0 memory.used_bytes=1073741824 memory.free_bytes=41264611328

A handful of identity / topology fields are baked onto the device at construction time and are not hot-reloadable in v1 — changing them requires a pod restart or a Helm change:

device name
architecture
brand
compute_capability
uuid
PCI bus_id
the BAR1 aperture — bar1_memory (baked at construction, so e.g. set --gpu 0 bar1_memory.total_bytes=... won't take effect until a pod restart)

If you need to change any of those, edit the profile / Helm values and restart the DaemonSet (or the affected pod).

For which reported values are simulated, which are static, and which are fixed by design — including why DCGM_FI_PROF_* does not track real work, and why no value responds to a pod holding an nvidia.com/gpu claim — see Metric Fidelity.

Where it runs¶

nvml-mock-ctl ships inside the nvml-mock DaemonSet image and runs via kubectl exec into the DaemonSet pod on the target node. Its scope is per-node: it only affects the node whose pod you exec into. To change several nodes, repeat the command against each node's pod.

Inside the pod the config path is wired through MOCK_NVML_CONFIG, and the config override defaults to the host-mounted driver config dir (/var/lib/nvml-mock/driver/config/overrides.yaml), so you normally run the subcommands with no path flags. (Set MOCK_NVML_OVERRIDES / --file only to override that default.)

# Pick the nvml-mock pod on a specific node.
# Replace <node> with the node name; adjust -n if you installed elsewhere.
POD=$(kubectl -n mokka get pod -l app.kubernetes.io/name=nvml-mock \
  --field-selector spec.nodeName=<node> -o jsonpath='{.items[0].metadata.name}')

Command reference¶

nvml-mock-ctl --help lists the commands, and nvml-mock-ctl <command> --help documents that command's own flags, defaults and positional arguments.

NAME:
   nvml-mock-ctl - mutate the simulated GPU state of a running nvml-mock node

USAGE:
   nvml-mock-ctl [global options] [command [command options]]

COMMANDS:
   fail               inject a device failure, or clear one with --mode healthy
   temp, temperature  pin reported GPU temperature
   power              pin reported power draw
   fan                pin reported fan speed (forces fan count >= 1)
   util, utilization  pin reported GPU + memory utilization
   clocks             pin reported SM + graphics clocks
   throttle           set the active throttle reasons ('none' clears them)
   pstate             pin the reported performance state (P-state)
   nvlink-error       inject NVLink DL errors at a rate in errors/second (0 heals)
   sram-ecc           inject SRAM ECC errors (0 heals)
   fabric-health      degrade NVLink fabric health ('healthy' clears it)
   set                write arbitrary schema fields, as key.path=value
   status             print the overrides currently in effect
   reset              clear the targeted overrides, returning the device(s) to the pristine profile
   watch-allocations  mirror each pod's nvidia.com/gpu claim into memory.used_bytes/free_bytes
   help, h            Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --file string    config override path (default: "/var/lib/nvml-mock/driver/config/overrides.yaml") [$MOCK_NVML_OVERRIDES]
   --config string  config path for UUID resolution and validation (default: "/var/lib/nvml-mock/driver/config/config.yaml") [$MOCK_NVML_CONFIG]
   --help, -h       show help

Every mutating command takes --gpu <idx|all|uuid>; reset applies to every device without it, and status reports every override. The global flags are accepted on either side of the command, but --gpu and the command's own flags must follow it.

nvml-mock-ctl fail --gpu <t> --mode <healthy|lost|fallen_off_bus|ecc_uncorrectable> [--after-calls N] [--xid CODE]
nvml-mock-ctl temp --gpu <t> celsius
nvml-mock-ctl power --gpu <t> watts
nvml-mock-ctl fan --gpu <t> percent
nvml-mock-ctl util --gpu <t> percent
nvml-mock-ctl clocks --gpu <t> mhz
nvml-mock-ctl throttle --gpu <t> reason [reason ...]
nvml-mock-ctl pstate --gpu <t> pstate
nvml-mock-ctl nvlink-error --gpu <t> [--links a,b,c] errors_per_sec
nvml-mock-ctl sram-ecc --gpu <t> [--type correctable|parity|secded]
              [--source l2|sm|microcontroller|pcie|other] [--threshold-exceeded] count
nvml-mock-ctl fabric-health --gpu <t> condition [condition ...]
nvml-mock-ctl set --gpu <t> key.path=value [key.path=value ...]
nvml-mock-ctl status [--gpu <idx>]
nvml-mock-ctl reset [--gpu <t>]
nvml-mock-ctl watch-allocations [--socket PATH] [--interval D] [--used-fraction F]

Targeting: `--gpu <idx|all|uuid>`¶

index — a device index, e.g. --gpu 0.
all — the shared all bucket; applies to every device unless a per-device override wins.
UUID — a GPU UUID, e.g. --gpu GPU-12345678-.... UUID targeting resolves the UUID to an index using the profile config, so it only works for devices that declare an explicit uuid: in the YAML. Devices whose UUID is auto-generated (no uuid: in the profile) cannot be targeted by UUID — use the index instead.

`fail` — inject or clear a failure¶

Sets the failure block for the target. Modes:

mode	effect
`healthy`	removes the failure override (recovers the device)
`lost`	guarded calls, handle lookups, and event-set waits return `ERROR_GPU_IS_LOST`
`fallen_off_bus`	same surface as `lost` (models a GPU that fell off the PCIe bus)
`ecc_uncorrectable`	device stays addressable; uncorrectable ECC counters climb

--after-calls N — trip deterministically after N guarded NVML calls (omit for "trip on first guarded call").
--xid CODE — surface this Xid code through the NVML event set once the device trips (delivered for any tripped failure mode with a Xid configured, e.g. --mode ecc_uncorrectable --xid 79), and have the node agent announce it on the node's kernel log — see below.

Xid on the kernel log¶

A real driver raises an Xid with a kernel printk, and the agents that watch for Xids read that line rather than NVML: some scan /dev/kmsg directly, while NVSentinel's syslog monitor reads the journal, admitting only kernel-transport entries. So an injection that carries an --xid also reaches the kernel log, once per targeted GPU:

kernel: NVRM: Xid (PCI:0000:1a:00): 79

The node agent writes it, not this CLI. The agent watches the same override document fail writes and announces each Xid that appears on it, so the kernel log follows the simulated GPU state itself rather than one way of changing it — an injection from the allocation watcher, a hand-edited override or a future control plane raises the line just the same. It also means the announcement is the node's to make: the agent names each device by the address it serves through NVML, and only the devices this node actually runs (gpu.count), which the CLI could only infer from the profile.

That address — the profile's pci.bus_id, or the mock's own for a profile that sets none — is spelled the way the driver spells it, lower case and without the function, so the kernel line, the NVML event and the simulated /sys/bus/pci tree all name the same device. Recovery announces nothing: --mode healthy clears the injection, but kernel logs never retract an Xid. Re-injecting after a recovery announces again, even with the same code, because it is a new fault.

The line appears within about a second of the injection, alongside the NVML event: both sides poll the override document. With --after-calls N the kernel line leads the NVML failure, because a deferred failure trips inside whichever consumer makes the Nth guarded call — a counter that lives in that process and never reaches disk, so nothing outside it can wait for the trip. Omit --after-calls when a consumer must not see the Xid before NVML reports the device failed.

Announcing is best-effort and never fails the injection: a node whose kernel log is missing or unwritable keeps the NVML side, and the agent logs the skip.

Two node-level requirements, neither of which the mock can arrange for itself:

the agent must be able to write /dev/kmsg, which takes nodeAgent.kernelLog.enabled=true. It is off by default because it mounts the device and runs node-agent privileged: mounting alone is not enough, as the container device cgroup rejects the write (operation not permitted) even for root with the node's world-writable /dev/kmsg bind-mounted, and no lesser capability lifts that. A cluster that will not take a privileged pod, or a node with no /dev/kmsg, has to leave it off; the DaemonSet then sets MOCK_NVML_KMSG="" and the agent stays quiet instead of warning about a device it was not given. local/nvml-mock.values.yaml and the nv-sentinel demo enable it, so an Xid injected there lands on the kernel log. Where PodSecurity is enforced, the namespace has to admit the pod: kubectl label namespace mokka pod-security.kubernetes.io/enforce=privileged — under baseline or restricted the DaemonSet is rejected outright rather than degraded;
for journal-based consumers, journald must ingest the kernel ring buffer (ReadKMsg=yes) and keep the journal where the consumer looks. Kind's node image sets ReadKMsg=no and keeps a volatile journal, so a Kind cluster needs a journald drop-in with Storage=persistent and ReadKMsg=yes. Applied after boot it takes two restarts — systemd-journald cannot reload, and it does not create /var/log/journal itself; systemd-journal-flush is what creates the directory and moves the journal there.

On a Kind cluster the kernel line is not node-local. The kernel ring buffer is not namespaced, so every node container on a host shares the machine's: an Xid written on one node is in dmesg on all of them, and on the host. Ingest it on one node only (the NVSentinel demo's run.sh configures a single worker), or every node's monitor reports the same fault and a remediator quarantines nodes whose GPUs are fine. Real nodes each have their own kernel, so this is an artifact of sharing a host, not something a consumer needs to handle.

fail --mode healthy is how you recover a single device (it deletes the failure block from that bucket). See the failure-injection section of the mock NVML README for the full per-mode semantics.

`temp` / `power` / `fan` / `util` / `clocks` / `throttle` / `pstate` — pin a common metric¶

These convenience commands pin the most-tweaked readings to a fixed value. Except for throttle they take a single positional argument:

command	argument	pins
`temp --gpu <t> <celsius>`	0–200 °C	`nvidia-smi ... temperature.gpu`
`power --gpu <t> <watts>`	watts (≥0)	`nvidia-smi ... power.draw` (converted to mW)
`fan --gpu <t> <percent>`	0–100 %	`nvidia-smi ... fan.speed`
`util --gpu <t> <percent>`	0–100 %	`nvidia-smi ... utilization.gpu,utilization.memory`
`clocks --gpu <t> <mhz>`	0–100000 MHz	`nvidia-smi ... clocks.sm,clocks.gr`
`throttle --gpu <t> <reason>...`	reason name(s)	`nvidia-smi ... clocks_throttle_reasons.*`
`pstate --gpu <t> <0-15>`	P-state number	`nvidia-smi ... pstate`

They exist because pinning these fields by hand is fiddly (see Dynamic metrics mask their static counterparts below). Each command does the right thing regardless of how the profile is configured:

temp and power write both the static block and a zero-variation dynamic block (ramp_c/variance_c = 0, variance_mw = 0), so the reading is deterministic whether or not the profile runs the dynamic-metrics simulator. The engine rebuilds the simulator on the next TTL, so running consumers converge without a restart. power accepts watts (the unit nvidia-smi displays) and converts to the milliwatts NVML uses; the value is still clamped to the profile's [min_limit_mw, max_limit_mw] envelope.
fan sets fan.speed_percent and forces fan.count to at least 1 (a larger baseline count is preserved). Liquid/passively-cooled profiles ship fan.count: 0, which makes fan.speed report [N/A]; forcing the count makes the pinned speed observable. There is no dynamic fan simulator, so this touches only the static fan block.
util pins GPU and memory utilization to the same percent. It sets the static utilization block and disables the dynamic utilization sub-simulator (writes dynamic_metrics.utilization: null), so the value is deterministic for any percent — including 0, which a zero-variation dynamic block could not express (the simulator treats min==max==0 as "unbounded").
clocks pins the reported SM and graphics clocks (clocks.sm_current and clocks.graphics_current). There is no dynamic clock simulator, so it hot-reloads directly. Memory/video clocks keep their profile baseline — use set clocks.memory_current=<mhz> to change those.
throttle sets the active clock-throttle reasons. It is authoritative: the requested reasons are turned on and every other reason is turned off, so repeated calls replace (not accumulate) state. Pass one or more reason names, or none (on its own) to clear them all. Accepted names are the clocks_throttle_reasons field names plus short aliases: thermal (hw_thermal_slowdown), sw_thermal (sw_thermal_slowdown), power (sw_power_cap), power_brake (hw_power_brake_slowdown), idle (gpu_idle), app_clocks (applications_clocks_setting), display_clocks (display_clocks_setting), plus hw_slowdown and sync_boost.
pstate pins the performance state to P<n> for n in 0–15.

reset clears these overrides and returns the metric to the profile baseline (varying again, if the profile drives it dynamically). For anything these commands don't cover, use set below.

`nvlink-error` — inject NVLink DL errors on switch links¶

Injects a per-link NVLink data-link error accrual on the target device's links to its NVSwitch, so the GPU's uplinks report climbing DL errors. The positional argument is the error rate in errors/second (0–1e9); 0 heals (no injection).

# ramp 250 NVLink errors/sec on every active link of GPU 0
kubectl -n mokka exec "$POD" -- nvml-mock-ctl nvlink-error --gpu 0 250
# restrict to specific link ids
kubectl -n mokka exec "$POD" -- nvml-mock-ctl nvlink-error --gpu 0 250 --links 0,3,7
# heal
kubectl -n mokka exec "$POD" -- nvml-mock-ctl nvlink-error --gpu 0 0

The count climbs monotonically off the shared counter epoch (the same accrual model as the profile's nvlink.defaults.error_rate), so consumers that sample over time see a rising error rate rather than a one-shot step. It surfaces on both the per-counter direct API (nvmlDeviceGetNvLinkErrorCounter) and the DL error field values (NVML_FI_DEV_NVLINK_ERROR_DL_{REPLAY,RECOVERY,CRC}, field ids 161–163, e.g. dcgmi dmon -e 161,162,163). Injection lands only on links the device actually has active — a nonexistent link is never conjured into an errored one — and --links (comma-separated ids) narrows it further; omit it to target all active links (the "GPU lost its switch uplinks" fault).

Why this and not an "NVSwitch health" injection? DCGM's NVSwitch entity health (DCGM_HEALTH_WATCH_NVSWITCH_*) and SXID errors are sourced from NSCQ (libnvidia-nscq.so) and kernel logs, not NVML, so a libnvidia-ml.so mock cannot drive them. The GPU-side NVLink error surface this command feeds is what DCGM's DCGM_HEALTH_WATCH_NVLINK reads (→ DCGM_FR_NVLINK_*), which is the switch-link fault NVSentinel's gpu-health-monitor can actually detect and remediate.

`sram-ecc` — inject on-die SRAM ECC errors¶

Sets the target's ecc.sram counters, so the GPU reports SRAM errors the way one that has just taken an SRAM fault does. The positional argument is the error count (0–1e9); 0 heals.

# 4 uncorrectable SEC-DED errors on GPU 0's SM, past the service threshold
kubectl -n mokka exec "$POD" -- nvml-mock-ctl sram-ecc --gpu 0 \
  --type secded --source sm --threshold-exceeded 4
# correctable errors (no source attribution on real hardware)
kubectl -n mokka exec "$POD" -- nvml-mock-ctl sram-ecc --gpu 0 --type correctable 12
# heal
kubectl -n mokka exec "$POD" -- nvml-mock-ctl sram-ecc --gpu 0 0

flag	meaning
`--type`	`correctable`, `parity` or `secded` (uncorrectable SEC-DED, the default)
`--source`	the unit the uncorrectable errors are attributed to; defaults to `other`
`--threshold-exceeded`	raises `SRAM Threshold Exceeded`, the driver's "this GPU needs servicing" signal

The count lands in both the volatile and aggregate scopes: hardware that has just taken an SRAM fault reports it in both, and pinning only one would leave a GPU whose history disagrees with its present. Within ecc.sram the command is authoritative — every counter and source is written — so repeated calls replace rather than accumulate, and --source moves the whole count to the named unit. The ECC mode is left alone, since a GPU with ECC off reports no SRAM counters at all and the injection would erase itself.

Values surface in nvidia-smi -q under ECC Errors (both scopes, SRAM Threshold Exceeded and Aggregate Uncorrectable SRAM Sources) and through the per-location field values DCGM reads.

`fabric-health` — degrade NVLink fabric health¶

Degrades the target's NVLink fabric health — the block nvidia-smi -q renders under Fabric → Health, and the first thing a fault-handling controller reads to decide whether a GPU's fabric attachment is usable. Positional arguments name the conditions to report:

# a route on GPU 0 is unhealthy
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fabric-health --gpu 0 route_unhealthy
# degraded fabric bandwidth while a route recovers
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fabric-health --gpu 0 degraded_bandwidth route_recovery
# the fabric manager gave this GPU no partition
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fabric-health --gpu 0 no_partition
# recover
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fabric-health --gpu 0 healthy

condition	reported as
`degraded_bandwidth`	`Bandwidth: Degraded`, `Summary: Limited Capacity`
`route_recovery`	`Route Recovery in progress: True`
`route_unhealthy`	`Route Unhealthy: True`
`access_timeout_recovery`	`Access Timeout Recovery: True`
`no_partition`, `insufficient_nvlinks`, `incompatible_gpu_fw`, `invalid_location`, `incorrect_sysguid`, `incorrect_chassis_sn`, `gpu_state_invalid`	`Incorrect Configuration: <that misconfiguration>`
`healthy`	clears every condition

Like throttle, it is authoritative: the named conditions are turned on and every other condition is turned off, so repeated calls replace rather than accumulate. healthy (on its own) clears them all — recovery does not need reset, so the device's other overrides survive it.

The Summary row follows the conditions (Unhealthy for a fault, Limited Capacity for degraded bandwidth alone, Healthy when clear), including for a profile that pinned fabric.health_summary: injecting a fault releases the pin, because a summary that stays Healthy while a condition reports a fault is not a state hardware can be in. See fabric health configuration for the static equivalent.

`set` — set arbitrary fields¶

set takes one or more key.path=value pairs. The path is the YAML/JSON path into the device config; the value is parsed as a YAML scalar (so numbers, bools, and strings get their natural type). Example paths: thermal.temperature_gpu_c, utilization.gpu, ecc.mode_current, power.current_draw_mw, remapped_rows.availability_histogram.low, clocks_throttle_reasons.counters.sw_power_cap_us.

Dynamic metrics mask their static counterparts¶

When the profile enables dynamic metrics (gpu.dynamicMetrics.enabled=true, which the demo/e2e charts do), the simulator drives temperature, power, and utilization and masks the static blocks for those fields. In that mode:

set thermal.temperature_gpu_c=<n> has no visible effect — the simulator keeps producing its own reading.
To pin temperature, override the dynamic block and zero its variation so the reading is deterministic:

nvml-mock-ctl set --gpu 0 \
  dynamic_metrics.temperature.base_c=85 \
  dynamic_metrics.temperature.ramp_c=0 \
  dynamic_metrics.temperature.variance_c=0

The engine rebuilds the simulator on the next TTL, so a running consumer sees the pinned value without a restart; reset returns it to the varying baseline. The temp command above does exactly this for you — prefer it (and the power/util commands) unless you need a field the convenience commands don't cover.

If dynamic metrics is disabled, the static thermal.temperature_gpu_c (and power.*, utilization.*) are authoritative and hot-reload directly. The same masking applies to power (dynamic_metrics.power) and utilization (dynamic_metrics.utilization). Note util handles this by disabling the dynamic utilization sub-simulator rather than zeroing its variation, so it pins correctly even at 0%.

To change several fields at once, pass multiple key.path=value pairs to a single set invocation.

`status` — inspect active overrides¶

status prints the current overrides.yaml. status --gpu <idx> filters to a single device's bucket plus the shared all bucket. status --gpu only accepts an integer index (not all or a UUID). With no active overrides it prints no active overrides.

`reset` — remove overrides¶

reset --gpu <t> removes the override bucket for the target. reset with no --gpu clears everything (equivalent to reset --gpu all). State reverts to the pristine profile within one TTL.

Reset via `nvidia-smi`¶

nvidia-smi --gpu-reset (-r) performs the same reset, so a remediation controller or a runbook that already reaches for the standard GPU reset works unmodified against the mock:

kubectl -n mokka exec "$POD" -- nvidia-smi -r -i 0   # same as: reset --gpu 0
kubectl -n mokka exec "$POD" -- nvidia-smi -r        # every GPU, one at a time

It reports per GPU (GPU 00000000:0A:00.0 was successfully reset.) and exits 0.

Two differences from the CLI follow from nvidia-smi driving the reset one device at a time:

A bare -r resets each GPU individually, so it does not clear the shared all: bucket. State injected with --gpu all needs nvml-mock-ctl reset --gpu all.
Resetting a GPU that has nothing injected leaves overrides.yaml untouched (no write, no lock), so a reset on a healthy GPU never depends on the overrides being writable.

The reset works from an injected consumer container as well as from the nvml-mock pod, which is what lets a remediation controller run it where the workload sees the GPU. Clearing a bucket rewrites overrides.yaml under an flock, so the config directory is bind-mounted writable into injected containers (the driver library and nvidia-smi itself stay read-only). Mounted read-only it would fail with GPU Reset couldn't run on exactly the GPUs that had state to clear, while healthy ones reported success through the no-write path above.

Reset semantics¶

Action	Effect on runtime overrides	Result
`nvml-mock-ctl reset [--gpu <t>]`	clears the targeted bucket(s) from `overrides.yaml`	device(s) revert to pristine profile within one TTL
`nvidia-smi --gpu-reset [-i <idx>]`	clears the same per-device bucket(s) — see via nvidia-smi	device(s) revert to pristine profile within one TTL
`nvml-mock-ctl fail --gpu <t> --mode healthy`	removes just the `failure` block for the target	that device recovers within one TTL; other overrides stay
`nvml-mock-ctl fabric-health --gpu <t> healthy`	clears just the fabric health conditions for the target	that device's fabric reports healthy within one TTL; other overrides stay
DaemonSet pod restart	the node daemon deletes `overrides.yaml` at startup	all overrides wiped; back to pristine profile
Consumer pod restart	none — the config override lives on the node, not in the consumer	consumer re-reads and picks up the current config override (does not reset it)
`helm upgrade` (profile/values change)	rolls the DaemonSet pod (config checksum + `RollingUpdate`), so the node daemon wipes `overrides.yaml` on the new pod	all overrides reset to the new pristine config; only an upgrade that does not recreate the nvml-mock pod leaves an config override in place

Worked examples¶

All examples assume $POD is set as shown in Where it runs.

# 1) Force uncorrectable ECC on GPU 0, deliver Xid 79
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fail --gpu 0 --mode ecc_uncorrectable --after-calls 1 --xid 79
# verify from any consumer pod:
kubectl exec <consumer> -- nvidia-smi --query-gpu=ecc.errors.uncorrected.aggregate.total --format=csv,noheader
# the same Xid on the node's kernel log (on Kind, read it on the node itself):
docker exec <node> sh -c 'dmesg | grep "NVRM: Xid"'

# 2) Mark ALL GPUs lost
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fail --gpu all --mode lost

# 3) Pin GPU 3's reported temperature to 85 C (works whether or not the
# profile drives temperature dynamically — the temp command handles both).
kubectl -n mokka exec "$POD" -- nvml-mock-ctl temp --gpu 3 85
# verify from any consumer pod:
kubectl exec <consumer> -- nvidia-smi --id=3 --query-gpu=temperature.gpu --format=csv,noheader,nounits

# 3b) Pin power draw to 350 W on all GPUs, and fan speed to 60% on GPU 0.
kubectl -n mokka exec "$POD" -- nvml-mock-ctl power --gpu all 350
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fan --gpu 0 60
# verify from any consumer pod:
kubectl exec <consumer> -- nvidia-smi --query-gpu=index,power.draw,fan.speed --format=csv,noheader

# 3c) Pin utilization, clocks, a throttle reason and the P-state on GPU 0.
kubectl -n mokka exec "$POD" -- nvml-mock-ctl util --gpu 0 90
kubectl -n mokka exec "$POD" -- nvml-mock-ctl clocks --gpu 0 1200
kubectl -n mokka exec "$POD" -- nvml-mock-ctl throttle --gpu 0 thermal
kubectl -n mokka exec "$POD" -- nvml-mock-ctl pstate --gpu 0 8
# verify from any consumer pod:
kubectl exec <consumer> -- nvidia-smi --id=0 \
  --query-gpu=utilization.gpu,clocks.sm,clocks_throttle_reasons.hw_thermal_slowdown,pstate \
  --format=csv,noheader
# clear the throttle reason again:
kubectl -n mokka exec "$POD" -- nvml-mock-ctl throttle --gpu 0 none

# 3d) Degrade GPU 0's NVLink fabric route while the workload keeps running.
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fabric-health --gpu 0 route_unhealthy
# verify from any consumer pod (Fabric -> Health block):
kubectl exec <consumer> -- nvidia-smi --id=0 -q | grep -A 7 Health
# recover, keeping the device's other overrides:
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fabric-health --gpu 0 healthy

# 4) Set several fields on GPU 0 in one call
kubectl -n mokka exec "$POD" -- nvml-mock-ctl set --gpu 0 \
  ecc.mode_current=disabled \
  utilization.gpu=100

# 5) Target by UUID (requires an explicit uuid: in the profile for that device)
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fail --gpu GPU-12345678-1234-1234-1234-123456780000 --mode fallen_off_bus

# 6) Inspect active overrides
kubectl -n mokka exec "$POD" -- nvml-mock-ctl status
# or just GPU 0 (integer index only):
kubectl -n mokka exec "$POD" -- nvml-mock-ctl status --gpu 0

# 7) Recover one GPU, then reset everything
kubectl -n mokka exec "$POD" -- nvml-mock-ctl fail --gpu 0 --mode healthy
kubectl -n mokka exec "$POD" -- nvml-mock-ctl reset --gpu all

# 8) Full reset via pod restart (the node daemon wipes overrides.yaml at startup)
kubectl -n mokka delete pod "$POD"

Troubleshooting¶

Changes aren't visible immediately. Propagation is bounded by the config override TTL (~1s default, MOCK_NVML_OVERRIDES_TTL). Wait one TTL and re-check. nvidia-smi spawns a fresh process on every call, so it always reflects the current config override once the TTL has elapsed; long-lived in-process NVML clients pick up the change on their next getter after the TTL.
Confirm what's actually applied. Run nvml-mock-ctl status (optionally --gpu <idx>), or read overrides.yaml directly on the node.
The command was rejected. Unknown fields or bad value types fail the write (nothing is applied). Check the field path against the profile schema in the Configuration Reference — e.g. it's thermal.temperature_gpu_c, not temperature.gpu_temp_c.
UUID target won't resolve. The device likely has an auto-generated UUID (no uuid: in the profile). Target it by index instead.
Nothing changed on other nodes. Scope is per-node. Repeat the command against each node's DaemonSet pod.
The Xid isn't in the kernel log. Silence usually means the announcement was never turned on: it needs nodeAgent.kernelLog.enabled=true, and without it the DaemonSet sets MOCK_NVML_KMSG="". Otherwise the node agent's log says why — it writes the line, so that is where the failure is reported: kubectl -n mokka logs "$POD" -c node-agent | grep kernellog. A failed write means the container cannot reach /dev/kmsg. If dmesg shows the line but a journal consumer does not see it, the node's journald is dropping kernel messages — see Xid on the kernel log.
An identity field didn't change. Device name, architecture, brand, compute_capability, uuid, and PCI bus_id are baked at construction and are not hot-reloadable in v1. Change the profile/Helm values and restart the pod.

Runtime control with nvml-mock-ctl¶

How it works¶

v1 scope — what is and isn't hot-reloadable¶

Where it runs¶

Command reference¶

Targeting: --gpu <idx|all|uuid>¶

fail — inject or clear a failure¶

Xid on the kernel log¶

temp / power / fan / util / clocks / throttle / pstate — pin a common metric¶

nvlink-error — inject NVLink DL errors on switch links¶

sram-ecc — inject on-die SRAM ECC errors¶

fabric-health — degrade NVLink fabric health¶

set — set arbitrary fields¶