Configuration Reference¶
Complete guide to Mock NVML configuration options.
Configuration Methods¶
Mock NVML supports two configuration methods:
| Method | Use Case | Flexibility |
|---|---|---|
| YAML File | Full control, per-device settings | High |
| Environment Variables | Simple scenarios, CI/CD | Low |
YAML configuration takes precedence when MOCK_NVML_CONFIG is set.
Environment Variables¶
| Variable | Description | Default |
|---|---|---|
MOCK_NVML_CONFIG |
Path to YAML configuration file | (none) — the library then tries, in order, config/config.yaml in the driver root it was loaded from, the CDI mount /etc/nvml-mock/config.yaml and the NRI mount /opt/nvml-mock/driver/config/config.yaml |
MOCK_MIG_PROFILES_CONFIG |
Path to the board's MIG profile table, which lives in a document of its own | (none) — a sibling of MOCK_NVML_CONFIG is tried instead |
MOCK_NVML_NUM_DEVICES |
Number of GPUs to simulate | 8 |
MOCK_NVML_DRIVER_VERSION |
NVIDIA driver version string | 550.163.01 |
MOCK_NVML_DEBUG |
Enable debug logging (any value) | (disabled) |
Example:
export MOCK_NVML_NUM_DEVICES=4
export MOCK_NVML_DRIVER_VERSION=550.163.01
LD_LIBRARY_PATH=. nvidia-smi
YAML Configuration Structure¶
version: "1.0" # Config format version (required)
system: # System-level settings
driver_version: "550.163.01" # Required; also gates which NVML functions exist
nvml_version: "12.550.163.01"
cuda_version: "12.4"
cuda_version_major: 12
cuda_version_minor: 4
driver_branch: "r550_00" # Optional; derived from driver_version when unset
hic: # Optional; host interface cards `nvidia-smi -q -u` prints
- id: 0
firmware_version: "1.2.3.4"
device_defaults: # Default settings for all devices
name: "NVIDIA A100-SXM4-40GB"
# ... (see Device Properties below)
devices: # Per-device overrides
- index: 0
uuid: "GPU-12345678-1234-1234-1234-123456780000"
pci:
bus_id: "0000:07:00.0"
- index: 1
uuid: "GPU-12345678-1234-1234-1234-123456780001"
pci:
bus_id: "0000:0F:00.0"
# ...
nvlink: # Optional NVLink configuration
version: 4
links_per_gpu: 18
Device Properties¶
Basic Identification¶
device_defaults:
name: "NVIDIA A100-SXM4-40GB" # GPU name shown in nvidia-smi
brand: "nvidia" # nvidia, tesla, quadro, geforce
serial: "1234567890123" # Serial number
board_part_number: "900-21001-0000-000"
vbios_version: "92.00.45.00.03"
hostname: "gpu-node-0" # nvmlDeviceGetHostname_v1; Blackwell or newer
Architecture¶
device_defaults:
architecture: "ampere" # kepler, maxwell, pascal, volta, turing,
# ampere, ada, hopper, blackwell, rubin
compute_capability:
major: 8
minor: 0
num_gpu_cores: 6912
Names are matched case-insensitively with surrounding whitespace trimmed, and
ada_lovelace is accepted as a spelling of ada. An unrecognized name is
logged as a warning and reported as NVML_DEVICE_ARCH_UNKNOWN, which fails
every architecture-gated feature — T.Limit temperatures and GPM support among
them.
Memory¶
device_defaults:
memory:
total_bytes: 42949672960 # 40 GiB
reserved_bytes: 0
free_bytes: 42949672960
used_bytes: 0
bar1_memory:
total_bytes: 68719476736 # 64 GiB
free_bytes: 68719476736
used_bytes: 0
PCI/PCIe¶
device_defaults:
pci:
device_id: 0x20B010DE # A100 device ID
subsystem_id: 0x134F10DE
bus_id: "0000:07:00.0" # Usually per-device
pcie:
max_link_gen: 4
current_link_gen: 4
max_link_width: 16
current_link_width: 16
replay_counter: 0
tx_throughput_kbps: 0
rx_throughput_kbps: 0
lspci sees these through the rendered sysfs tree and enumerates every GPU, but
two of its lines are missing next to real hardware. Kernel driver in use:
nvidia needs a driver symlink, which the tree does not render and the
libmockfs.so shim could not surface anyway — it intercepts open/stat, not
readlink. Kernel modules: needs libkmod, which fails to initialise in a
container with no /lib/modules; that is where the Unable to load libkmod
resources: error -2 on lspci -v comes from, and a container on real hardware
prints it too. Both are display-only: lspci still exits 0, and a consumer
reading identity out of sysfs gets the full picture.
Platform identity (rack location)¶
Where the node's boards sit in a rack, which nvidia-smi -q renders as its
Platform Info block. Rack-scale fault correlation reads it to turn a GPU fault
into a physical location. Only gb200 and gb300 ship it; on every other
profile the whole block reads N/A, which is the correct reading for a board
whose platform reports no location.
device_defaults:
platform:
chassis_serial_number: "1822725100200" # serial of the chassis (the rack)
slot_number: 21 # absolute physical slot, switch trays included
tray_index: 11 # position among compute trays only
host_id: 1 # the OS domain within the tray — this node
peer_type: "switch_connected" # or "direct_connected"
devices:
- index: 0
platform:
module_id: 1 # ID of this GPU within the node
- index: 1
platform:
module_id: 2
Only module_id varies between the GPUs of a node: NVML defines it as the ID of
the GPU within the node, while the other fields describe the node, which occupies
exactly one tray in one slot of one chassis. All six are settable per device
anyway, and a device override merges field by field, so a device declares just
its module_id.
slot_number counts switch trays and compute trays alike while tray_index
counts compute trays only, so on an NVL72 rack — 18 compute trays, 9 switch
trays — slot runs ahead of tray for a tray above the switch group.
peer_type says how the GPU reaches its NVLink peers: switch_connected
(through an NVSwitch tray) renders as Switch Connected, and anything else,
including an absent key, renders as Direct Connected.
As everywhere else in a device override, a zero is indistinguishable from an
unset key, so module_id numbers from 1.
Declaring the block also requires a driver new enough to have the API:
nvmlDeviceGetPlatformInfo arrived in 560, so a profile on an older
system.driver_version reports N/A regardless. The GPU Fabric GUID row of
the same block is not modelled and reads 0x0000000000000000.
Power¶
device_defaults:
power:
management_supported: true
management_mode: "enabled"
default_limit_mw: 400000 # 400W
enforced_limit_mw: 400000
min_limit_mw: 100000 # 100W
max_limit_mw: 400000 # 400W
current_draw_mw: 72000 # 72W (idle)
power_state: "P0"
Thermal¶
device_defaults:
thermal:
temperature_gpu_c: 33 # Current temperature
temperature_memory_c: 31
shutdown_threshold_c: 92
slowdown_threshold_c: 87
max_operating_c: 83
target_temperature_c: 83
How these thresholds are reported depends on device_defaults.architecture,
as on real hardware. Ada and later expose them as signed T.Limit offsets
(nvmlDeviceGetMarginTemperature and the NVML_FI_DEV_TEMPERATURE_*_TLIMIT
field IDs), which nvidia-smi -q renders as the "T.Limit" rows. Pre-Ada
architectures report NVML_ERROR_NOT_SUPPORTED for both, so nvidia-smi -q
falls back to the absolute GPU Shutdown / Slowdown / Max Operating Temp rows.
Fan¶
device_defaults:
fan:
count: 0 # 0 = liquid cooled (A100 SXM)
speed_percent: "N/A"
target_speed_percent: "N/A"
count gates the whole family. A device that declares no dedicated fans answers
NVML_ERROR_NOT_SUPPORTED to every fan query, which nvidia-smi renders as
[N/A] — the behaviour of every profile shipped with Mokka, matching every
hardware capture in the repository, including the passively cooled L40S and T4
PCIe boards.
A device that does declare fans reports the rest of the block. The fan block is per-device as well as a default, so a profile can mix a fanned board with fanless modules:
devices:
- index: 0
fan:
count: 2
speed_percent: "40" # what the fans are running at
target_speed_percent: "45" # what the driver's curve is aiming for
max_speed_rpm: 5000 # tachometer reading at 100 %
min_speed_percent: 30 # floor a manual write may ask for
max_speed_percent: 90 # ceiling a manual write may ask for
speed_percent and target_speed_percent are strings because "N/A" is a
legitimate value: a board whose fans are driven but not reported. Anything
unparseable is treated the same way.
max_speed_rpm is scaled by the current duty cycle to answer
nvmlDeviceGetFanSpeedRPM, so the fans above read 2000 RPM. Leaving it unset
models a board whose fans are driven but not measured — the RPM reading is
refused while the percentage still answers, which is a split real hardware
shows.
min_speed_percent and max_speed_percent are the window
nvmlDeviceGetMinMaxFanSpeed advertises and the bound nvmlDeviceSetFanSpeed_v2
enforces; a write outside it is refused with NVML_ERROR_INVALID_ARGUMENT. An
unset ceiling reads as 100.
Fan control¶
Each fan follows either the driver's temperature curve or a duty cycle a
consumer asked for. controls seeds that state per fan, for a board that should
come up under manual control:
fan is the index NVML's per-fan entry points address, so an entry naming a fan
the device does not have is ignored rather than applied to one it does.
The four setters — nvmlDeviceSetFanSpeed_v2,
nvmlDeviceSetDefaultFanSpeed_v2, nvmlDeviceSetFanControlPolicy and, through
it, a policy switch — write to this state. Like the power management limit they
are recorded in the runtime override document rather than in the writing
process, so a later process reads the write back and it holds until a reset
(nvidia-smi -r, or nvml-mock-ctl reset). A mock with nowhere to record the
write refuses it with NVML_ERROR_NO_PERMISSION rather than reporting a success
nothing would observe.
Holding a fan at a duty cycle implies manual control, because a fan the driver
is still steering by temperature would not stay where it was put.
nvmlDeviceSetDefaultFanSpeed_v2 returns the fan to the curve and drops the
duty cycle with it. Switching to manual through nvmlDeviceSetFanControlPolicy
does not invent one: the fan keeps running where it was until something asks for
a different speed.
Only part of this is visible through nvidia-smi. Versions 575 and 580 call
just nvmlDeviceGetFanSpeed, which reports the board's first fan, so
--query-gpu=fan.speed and the <fan_speed> element are the whole fan surface
they expose: a manual write to fan 0 shows up there, and nothing else in this
section does. The remaining entry points answer a consumer holding NVML
directly.
nvml-mock-ctl fan is a separate convenience path. It pins speed_percent and
forces count to at least 1, so a speed becomes observable on a profile that
ships fanless.
S-class units¶
nvmlUnitGetFanSpeedInfo reports the fans of an S-class chassis. Units are the
management processors of the S870/S1070 generation; nothing Mokka models has
one, and nor does any GPU a modern driver sees, so the unit enumeration is
empty. nvidia-smi -q -u reports a node with no units rather than failing:
There is no configuration for units, and no handle the unit queries will accept.
Clocks¶
device_defaults:
clocks:
graphics_current: 210 # MHz
graphics_max: 1410
graphics_app: 1410
graphics_app_default: 1410
sm_current: 210
sm_max: 1410
memory_current: 1215
memory_max: 1215
memory_app: 1215
memory_app_default: 1215
video_current: 585
video_max: 1290
graphics_max answers two rows of nvidia-smi -q, not one: Max Clocks and Max
Customer Boost Clocks. NVML keeps them apart — the second is an OEM-defined
ceiling that may legally sit below the boost maximum — but every real board
captured in tests/e2e/go/assertions/nvidiasmi/testdata/hardware reports the two
as equal, so the mock resolves both from this key rather than carrying a second
one that profiles would have to keep in step by hand. A profile with no
graphics_max reports both rows as N/A.
Only the graphics domain answers the customer-boost row; nvidia-smi emits no
other child under it. nvmlDeviceGetClock resolves the whole clock-type ×
clock-id matrix from the keys above, and returns NVML_ERROR_NOT_SUPPORTED for
the combinations no key carries — application clocks for the SM and video
domains, and the OEM ceiling for anything but graphics.
Supported clocks¶
device_defaults:
supported_clocks:
memory_clocks: # highest first, as NVML returns them
- freq_mhz: 2619
graphics_clocks: [1980, 1965, 1950, 1935] # truncated; see h100.yaml
- freq_mhz: 1593
graphics_clocks: [1980, 1965, 1950, 1935]
The table is what nvidia-smi -q -d SUPPORTED_CLOCKS and
--query-supported-clocks print, and the set of pairs nvidia-smi -ac
accepts. The shipped profiles carry the full table their board reports in
tests/e2e/go/assertions/nvidiasmi/testdata/hardware, and a unit test keeps
them equal. A profile without the block reports the section as N/A.
The mock models one P-state: the device's performance_state. The table
supplies its clock envelope (nvmlDeviceGetMinMaxClockOfPState,
nvmlDeviceGetPerformanceModes), and any other P-state is reported as not
supported.
Tuning clocks at runtime¶
The tuning commands write to the runtime override document, the same way
nvidia-smi -pl does. Every process
on the node sees the result, and it holds until the matching reset command,
nvidia-smi -r, or nvml-mock-ctl reset.
| Command | Effect on the readings |
|---|---|
-ac <mem,graphics> / -rac |
Sets the applications clocks. A pair missing from supported_clocks is refused, and -rac restores the *_app_default pair |
-lgc <min,max> / -rgc |
Clamps the current graphics and SM clocks into the range |
-lmc <min,max> / -rmc |
Clamps the current memory clock into the range |
The locked ranges are stored as clocks.locked_graphics and
clocks.locked_memory (min_mhz, max_mhz), so a profile can also start a
device locked. -rgc and -rmc unlock the GPU even then, because unlocked
is the driver default. A full reset (nvidia-smi -r or nvml-mock-ctl
reset) drops every runtime write instead, so the profile's lock comes back.
NVML's symbolic bounds behave as in nvml.h: unlimited on both
ends unlocks, a mix of a symbolic bound and a MHz bound is refused, and the
TDP bound is not supported.
Clock offsets (nvmlDeviceSetClockOffsets and the *ClkVfOffset calls) are
opt-in, because no captured board reports them:
device_defaults:
clocks:
offsets:
graphics: {min_mhz: -200, max_mhz: 300}
memory: {min_mhz: 0, max_mhz: 1000}
adaptive_clocking: "enabled" # or "disabled"; absent = not supported
An offset write is range-checked and read back, but it does not move the
reported clocks. Auto boost (--auto-boost-default) is refused as not
supported, matching the N/A every Pascal-or-newer board reports.
device_defaults:
clocks_throttle_reasons:
gpu_idle: true
applications_clocks_setting: false
sw_power_cap: false
hw_slowdown: false
hw_thermal_slowdown: false
hw_power_brake_slowdown: false
sync_boost: false
sw_thermal_slowdown: false
display_clocks_setting: false
# Cumulative time each cause has already cost the GPU, in microseconds.
# The flags above answer "is it throttled now"; these answer "how long has
# it been", which is what a slow workload is diagnosed from. Omit the block
# for a GPU that has never been throttled: every counter then reports 0 us,
# which is an answer, where the pre-#678 N/A was not.
counters:
sw_power_cap_us: 39595
sync_boost_us: 0
sw_thermal_slowdown_us: 0
hw_thermal_slowdown_us: 0
hw_power_brake_slowdown_us: 0
A per-device clocks_throttle_reasons block replaces the device_defaults one
wholesale rather than merging into it, as memory, power, thermal and
clocks already do. A devices: entry carrying only counters therefore drops
the inherited flags, so restate the ones that still apply — which is why the
gpu_idle: true above is repeated in each per-device block of
tests/mocknvml/util-test-config.yaml.
The counters appear under Clocks Event Reasons Counters in nvidia-smi -q,
and are readable through nvmlDeviceGetFieldValues (the path DCGM uses) and
nvmlDeviceGetViolationStatus. A device accrues further time on top of its
configured baseline for as long as the matching flag is set. Accrual is per
process, like the dynamic-metrics simulator: a long-lived consumer such as
dcgm-exporter watches the counter climb, while each nvidia-smi invocation
accrues only over its own lifetime and so reports essentially the baseline.
That per-process scope is why the flags and the counters have to be configured
to agree rather than agreeing by construction. A GPU thrown into a throttle
state at runtime carries no history: nvidia-smi renders the reason Active
beside a counter of a microsecond or two, which is the age of that nvidia-smi
process and not of the throttle. To describe a GPU that has been throttling for
a while, seed the counter alongside the flag — both land in the same override
document, so the second command keeps the first:
nvml-mock-ctl throttle --gpu 3 thermal
nvml-mock-ctl set --gpu 3 clocks_throttle_reasons.counters.hw_thermal_slowdown_us=39595
Performance¶
device_defaults:
performance_state: "P0" # P0-P15
utilization:
gpu: 0 # 0-100%
memory: 0
encoder: 0
decoder: 0
jpeg: 0
ofa: 0
ECC¶
device_defaults:
ecc:
mode_current: "enabled"
mode_pending: "enabled"
default_mode: "enabled"
errors:
volatile:
single_bit:
device_memory: 0
l1_cache: 0
l2_cache: 0
register_file: 0
texture_memory: 0
total: 0
double_bit:
device_memory: 0
total: 0
aggregate:
single_bit:
total: 0
double_bit:
total: 0
SRAM ECC¶
Hardware counts on-die SRAM errors separately from the DRAM counters above and
reports them through their own API, so they are a sibling block rather than
another memory location under errors. Omitting the block reports zeros, which
is what a healthy ECC-enabled GPU does; with ECC off the counters are reported
unsupported (N/A), as on real hardware.
device_defaults:
ecc:
sram:
volatile: # reset when the driver reloads
correctable: 0
uncorrectable_parity: 0
uncorrectable_secded: 0
aggregate: # persisted in the InfoROM
correctable: 0
uncorrectable_parity: 0
uncorrectable_secded: 0
# Which unit reported the aggregate uncorrectable errors; nvidia-smi
# renders it as "Aggregate Uncorrectable SRAM Sources".
uncorrectable_sources:
l2: 0
sm: 0
microcontroller: 0
pcie: 0
other: 0
# Whether the accumulated errors passed the driver's threshold — the
# signal that the GPU needs servicing rather than just a count going up.
threshold_exceeded: false
Inject these at runtime with nvml-mock-ctl sram-ecc (see
nvml-mock-ctl.md).
How nvidia-smi renders these counters depends on the profile's
architecture, mirroring real hardware: Ampere and later split the uncorrectable
count into SRAM Uncorrectable Parity and SRAM Uncorrectable SEC-DED and print
the source breakdown and threshold flag, while pre-Ampere (t4) prints one
combined SRAM Uncorrectable row and omits the rest. The configuration is the
same either way — only the presentation differs.
Memory repair state¶
Health agents decide between a reset, a row remap and a replacement from three
queries: the repair status, the unrepairable-memory flag and the SRAM unique
uncorrectable error counts. They are derived from the state above and from
remapped_rows, so an injected fault moves all of them
together:
| Query | Reports | Architecture |
|---|---|---|
| Repair status | A channel repair is pending when remapped_rows.pending is set or an ecc_uncorrectable failure is injected. A TPC repair is never pending. |
Ampere and later |
| Unrepairable-memory flag | Set when remapped_rows.failure_occurred is set, an ecc_uncorrectable failure is injected, or ecc.sram.threshold_exceeded is set. |
Hopper and later |
| SRAM unique uncorrectable errors | One entry per non-zero ecc.sram.uncorrectable_sources unit, carrying the unit and its count. |
Blackwell and later, with ECC on |
Earlier architectures report each query as unsupported.
The same NVML calls write back. They persist in the runtime override document, so a separate process sees the result:
nvidia-smi -esetsmode_pendingonly;mode_currenttakes it when the GPU is reset withnvidia-smi -r, as on hardware.- Clearing the volatile or aggregate error counters zeroes the DRAM and SRAM
counters of that scope. Clearing the aggregate scope also zeroes
uncorrectable_sourcesandthreshold_exceeded, which are derived from the counters. - Setting the DRAM encryption mode writes
dram_encryption.mode_pending.
device_defaults:
ecc:
dram_encryption: # Blackwell and later; earlier parts report it unsupported
mode_current: "disabled"
mode_pending: "disabled"
The InfoROM validation query succeeds for any profile with an inforom block.
Its configuration checksum is a CRC-32 of the declared versions, so it is stable
for a profile and differs between profiles with different InfoROM versions.
Remapped rows¶
availability_histogram is how many memory banks still have spare rows to remap
future failures. Row remapping is Ampere and later, so leaving the block out —
as t4 does — reports the histogram as unsupported.
device_defaults:
remapped_rows:
correctable: 0
uncorrectable: 0
pending: false
failure_occurred: false
availability_histogram:
max: 640 # banks with full spare capacity
high: 0
partial: 0
low: 0
none: 0 # banks with no capacity left
Display¶
Modes¶
device_defaults:
persistence_mode: "enabled"
compute_mode: "default" # default, exclusive_thread,
# prohibited, exclusive_process
addressing_mode: "hmm" # hmm, ats, none
addressing_mode is the Addressing Mode row of nvidia-smi -q: how the GPU
reaches memory the host allocated with malloc or mmap. It belongs to the
host as much as the board: hmm needs kernel support, and ats needs a Grace
CPU translating over NVLink-C2C. The same GPU on the same driver can report
either, so each built-in profile declares the mode its captured node reported.
A devices[] entry can set it for one GPU. Leaving the key unset, or
misspelling it, renders N/A. The row needs a driver
of 580 or newer: on an older profile the query does not exist, as on real
hardware.
Drain, removal and exclusion¶
These model what the driver does with a GPU rather than the GPU itself: taking
a GPU out of service while the driver stays loaded, and a GPU the driver was
told never to manage. NVML addresses all of them by PCI address
(domain, bus and device; the function is ignored), and a GPU that matches none
returns NVML_ERROR_INVALID_ARGUMENT.
Exclusion is configuration. A device marked excluded, the equivalent of
listing it in the kernel module's NVreg_ExcludedGpus, is never enumerated:
the remaining GPUs are numbered as if it were absent. It is reported only by
nvmlGetExcludedDeviceCount and nvmlGetExcludedDeviceInfoByIndex, with its
PCI address and UUID, which is what nvidia-smi -B lists. Set it per device,
typically in a per-device override, or at runtime with
nvml-mock-ctl set --gpu 3 excluded=true:
Drain and removal are runtime state, set by a consumer through NVML and
recorded in the runtime override document as the
draining and removed fields. Like a power cap, they are visible to every
process on the node and hold until a reset clears them (nvml-mock-ctl reset).
The bundled nvidia-smi has no drain subcommand, so to drain a GPU without
writing an NVML client, set the field directly:
nvml-mock-ctl set --gpu 1 draining=true.
| Call | Effect | Refused with NVML_ERROR_IN_USE when |
|---|---|---|
nvmlDeviceModifyDrainState |
Marks the GPU draining, or clears the mark. A process that initialises afterwards does not enumerate a draining GPU; processes that already see it keep it. | Draining a GPU in persistence mode — run nvidia-smi -pm 0 first |
nvmlDeviceQueryDrainState |
Reports whether the GPU is draining. | — |
nvmlDeviceRemoveGpu |
Detaches the GPU from the driver. Every process stops enumerating it, the calling one included, and the rest are renumbered. The calling process's handles to the GPU, its MIG devices and instances return NVML_ERROR_INVALID_ARGUMENT from then on. |
The GPU is in persistence mode, or the device has configured processes |
nvmlDeviceDiscoverGpus |
Re-attaches removed GPUs: all of them when domain, bus and device are zero, otherwise the one at that address. As in NVML, the busId string is not read. A rediscovered GPU is no longer draining. |
— |
Note
A process that is already running sees a removal or rediscovery made by
another process only once it next calls nvmlInit. The process that made
the change sees it immediately.
MIG¶
Those three keys are the whole of device_defaults.mig, and they are all a
profile needs to describe what the silicon can do. The board's partition
table, where that table lives, and the two ways to declare a layout have a
page of their own: MIG.
Virtualization¶
device_defaults:
virtualization:
mode: "none" # none, passthrough, vgpu, host_vgpu
host_vgpu_mode: "N/A" # host_vgpu only: sriov, non_sriov
vgpu and host_vgpu simulate the two sides of NVIDIA vGPU and take a
virtualization.vgpu block; its fields, defaults and validation are on a page
of their own: vGPU.
InfoROM¶
device_defaults:
inforom:
image_version: "G500.0212.00.02"
oem_object: "2.0"
ecc_object: "6.16"
pwr_object: "1.0"
Accounting¶
Per-process accounting is what nvidia-smi --query-accounted-apps and DCGM's
job statistics read. With mode: "enabled" the buffer holds a record for every
configured process still running, plus the terminated records
you declare for processes that have already exited:
device_defaults:
accounting:
mode: "enabled" # enabled, disabled
buffer_size: 4000 # records kept; the oldest drop first
terminated: # oldest first
- pid: 3131
gpu_util: 62 # lifetime GPU utilization %
mem_util: 31 # lifetime memory utilization %
max_memory_mib: 1024
time_ms: 120000 # how long it ran
A running process's record takes its utilization and memory from its
processes entry and reports a run time of 0, as NVML does until a process
exits. Every process counts as started when the driver loaded. If a PID appears
twice, the later record wins, which matches NVML reusing a PID after the first
process exited.
With accounting disabled, and on a GPU with MIG enabled, the accounting queries
return NVML_ERROR_NOT_SUPPORTED. A PID with no record returns
NVML_ERROR_NOT_FOUND. nvmlDeviceGetAccountingStats_v2 needs a configured
driver of 600 or newer.
nvidia-smi -am (nvmlDeviceSetAccountingMode) and nvidia-smi -caa
(nvmlDeviceClearAccountingPids) are recorded in the
runtime override document, like a power cap. Clearing
drops the terminated records and keeps the running ones. Disabling accounting
drops the terminated records too, so they do not come back when it is enabled
again. Both calls are refused on a GPU with MIG enabled.
Encoder/Decoder¶
device_defaults:
encoder_stats:
session_count: 0
average_fps: 0
average_latency_us: 0
fbc_stats:
session_count: 0
average_fps: 0
average_latency_us: 0
Processes¶
processes drive the running-process queries, nvmlDeviceGetProcessUtilization,
nvmlDeviceGetProcessesUtilizationInfo and the running records in
accounting. The utilization fields
(sm_util/mem_util/enc_util/dec_util, all percent) are reported for every
process, whatever its type.
type decides which running-process query reports the process:
type |
Reported by |
|---|---|
C (or omitted) |
nvmlDeviceGetComputeRunningProcesses |
G |
nvmlDeviceGetGraphicsRunningProcesses |
M |
nvmlDeviceGetMPSComputeRunningProcesses only |
An M process is an MPS client. Under MPS (Multi-Process Service) a client's
work runs in the MPS server's context, so NVML lists the server among the
compute processes and the client only in the MPS list. To model an MPS node,
declare the server as a C process named nvidia-cuda-mps-server and its
clients as M. nvmlDeviceGetRunningProcessDetailList returns the same
processes as the query for its mode, or all of them for
NVML_PROCESS_MODE_ALL. The mock does not run an MPS daemon.
They also show up in nvidia-smi: the default table's Processes box, -q, and
--query-compute-apps. Two caveats there — nvidia-smi labels every row M+C+G
regardless of type, because the call it enumerates processes through carries no
type field, and nvidia-smi pmon uses a different entry point that the mock does
not implement, so it lists nothing.
device_defaults:
processes:
- pid: 1234
type: "C" # C=compute, G=graphics
name: "python"
used_memory_mib: 1024
sm_util: 75 # SM (compute) utilization %
mem_util: 40 # memory-bandwidth utilization %
- pid: 5678
type: "G"
name: "ffmpeg"
used_memory_mib: 512
enc_util: 60 # encoder utilization %
dec_util: 30 # decoder utilization %
- pid: 9012
type: "M" # MPS client
name: "train.py"
used_memory_mib: 4096
Per-Device Overrides¶
Override any property for specific devices:
devices:
- index: 0
uuid: "GPU-12345678-1234-1234-1234-123456780000"
minor_number: 0
pci:
bus_id: "0000:07:00.0"
# Override thermal for this device only
thermal:
temperature_gpu_c: 35
- index: 1
uuid: "GPU-12345678-1234-1234-1234-123456780001"
minor_number: 1
pci:
bus_id: "0000:0F:00.0"
thermal:
temperature_gpu_c: 37
index is the NVML enumeration position and the key the override matches on.
minor_number is the N in /dev/nvidia<N>, which the agent stages and both
CDI specs point at. It defaults to the index — the numbering a driver produces
when it probes in PCI enumeration order — so set it only to describe a node
where the two differ.
Minor numbers are a permutation of the device indices: remapping one device
means remapping whichever device held the minor it took. A profile that leaves
two devices on the same minor is rejected, defaults included, as is a minor
outside 0-254 (255 is nvidiactl).
NVLink Configuration¶
nvlink:
version: 4
links_per_gpu: 18
bandwidth_per_link_mbps: 26562
c2c_enabled: false
multi_node: false
nvle_enabled: false # see bandwidth mode, low power and NVLE
bw_mode: # see bandwidth mode, low power and NVLE
scope: system
links:
- link: 0
state: "active"
remote_device_type: "GPU"
remote_pci_bus_id: "0000:0F:00.0"
switches: # see NVSwitches below
- bdf: "0000:05:00.0"
uuid: "NVSwitch-H100-0000-0000-0000-000000000000"
device_id: 0x22A310DE
c2c_enabled is node-level and drives the GPU C2C Mode row of nvidia-smi -q
on every attached GPU: true reports Enabled, while false or an absent key
reports N/A — never Disabled, because NVML answers
NVML_ERROR_NOT_SUPPORTED, the correct reading for a board with no NVLink-C2C
link to a host CPU. Only gb200 and gb300 enable it. This is the only key that
drives the row: device_defaults.features.nvlink_c2c is descriptive metadata and
is not read.
multi_node marks a board whose NVLink domain spans nodes, as in a rack-scale
NVLink system, as opposed to an NVLink fabric that stays on one baseboard, such
as HGX H100 or B200. Of the shipped profiles, gb200 and gb300 set it; Vera
Rubin and NVL576 profiles would set it too. On such a node the
node agent stages the IMEX capability surface the DRA driver's ComputeDomain
plugin reads: a substitute /proc/devices entry, the fabric-imex-mgmt
capability file and the channel0 device node, the only channel the driver of
a real node creates on its own. The chart's imex.deviceNumbers values set
their device majors; a major the node's kernel already gives to another driver
is replaced by a free one from 384 to 511. A node whose surface cannot be
staged stays ready without IMEX, and the node agent logs why.
Bandwidth mode, low power and NVLE¶
Three optional node-level keys sit here. h100, b200, gb200 and gb300
declare bw_mode, and gb300 also declares firmware; an omitted key reports
the default below:
nvlink:
nvle_enabled: true # default false; the NVLE: row of `nvidia-smi nvlink --info`
bw_mode: # default: none; every bandwidth-mode call answers NOT_SUPPORTED
scope: device # system or device
supported: [0, 1, 2, 3, 4] # default: all five
mode: 0 # default: the best supported mode
firmware: # default: none, which renders as N/A
mse: "15:3:0"
NVLink Reduced Bandwidth Mode comes as two independent NVML function families,
and a real board answers on one and declines the other. bw_mode.scope picks
the one that answers; the other returns NVML_ERROR_NOT_SUPPORTED. A scope
that names neither is warned about and treated as an absent block.
scope: system |
scope: device |
|
|---|---|---|
| NVML | nvmlSystemGetNvlinkBwMode, nvmlSystemSetNvlinkBwMode |
nvmlDeviceGetNvlinkBwMode, nvmlDeviceSetNvlinkBwMode, nvmlDeviceGetNvlinkSupportedBwModes |
nvidia-smi |
nvlink -gBwMode, nvlink -sBwMode |
no flag reaches it — NVML callers only |
| Shipped profiles | h100 |
b200, gb200, gb300 |
Either way, supported is the set the setter accepts, and the reported mode is
the last value a setter recorded, else mode, else the best supported mode.
Choosing a scope for a new profile
The NVML reference documents the node-wide family for Hopper or newer and
the per-device family for Blackwell or newer. The nvidia-smi manual,
however, marks -gBwMode and -sBwMode deprecated for Blackwell, and a
real GB300 answers -gBwMode with Getting nvlink bandwidth mode is not
supported. A Hopper board therefore takes system, a Blackwell board
device, and a board before Hopper omits the block. On a device profile
the mode is visible to NVML callers only.
The nvidia-smi manual documents the five mode names and what each does, but
nothing states which index the NVML calls carry for a name. The mapping below
was established against the nvidia-smi the mock image bundles, and the set
stops at four because a sixth value would index past its name table.
| Value | Name | Effect |
|---|---|---|
| 0 | FULL |
all links at maximum bandwidth |
| 1 | OFF |
NVLink unused; GPU-to-GPU traffic inside the node goes over PCIe instead |
| 2 | MIN |
minimum speed |
| 3 | HALF |
about half of FULL |
| 4 | 3QUARTER |
about 75% of FULL |
A supported value above 4 is accepted, since the indices are the driver's and
a later one may define more, but it has no name to print and the profile is
warned about it. A mode outside the effective supported set is ignored with
a warning and the best supported mode is reported instead —
nvmlDeviceSetNvlinkBwMode answers NVML_ERROR_INVALID_ARGUMENT for that very
value, so honouring it would have the getter hand out a mode the setter refuses.
mode is an initial value only. Both setters record into the config override
document rather than into process memory, so a mode set by one process is read
by every other process on the node and outlives the writer. A node-wide write
also clears the per-device records, which would otherwise outrank it. Inspect
either with nvml-mock-ctl status.
firmware fills the Firmware Version: rows that nvlink --info prints under
NVLE:. Each value is major:minor:subMinor — the form nvidia-smi prints —
so a line read off real hardware transfers to a profile unchanged:
nvlink:
firmware:
mse: "15:3:0"
netir: "36:2014:4784"
netir_uphy: "7:0:0"
netir_cln: "0:10:0"
netir_dln: "0:10:0"
The five keys are the microcontrollers NVML defines a ucode type for
(NVML_NVLINK_FIRMWARE_UCODE_TYPE_MSE through ..._NETIR_DLN), and
nvidia-smi renders them in that order whatever order the profile lists it in.
A sixth is not expressible, which is deliberate: nvidia-smi abandons the
whole --info render on a ucode type it cannot label. Omit a key and
its row is left out; omit the block and the table is empty, which nvidia-smi
renders as Firmware Version: N/A. A value that is not three unsigned
colon-separated parts is dropped with a warning rather than failing the load.
gb300 ships the set above, read off a GB300 NVL; no other profile declares
one.
The two remaining surfaces follow what the profile declares, so a board without the hardware declines instead of answering:
| Surface | Requirement |
|---|---|
nvlink -gLowPwrInfo, -sLowPwrThres |
A GPU with an active NVLink. The threshold is 1-1023, or 0xFFFFFFFF to restore the driver default |
the NVLE: and Firmware Version: rows of nvlink --info |
Driver 580 or newer, on a board that has NVLink (links_per_gpu). Below 580 the function itself is absent, so nvidia-smi reports Function Not Found rather than an unsupported device. There is no architecture floor: a Hopper board prints the rows too, with NVLE off and no firmware |
NVSwitches¶
switches: lists the NVSwitches on the node. They are modelled as NVLink remote
endpoints: declaring them (with links_per_gpu > 0 and no explicit per-device
links:) is what makes the engine fan each GPU's links across them, which is
what nvidia-smi topo -m renders as NV<links_per_gpu> between every GPU pair.
NVML offers no per-switch API, so the nvmlUnit* chassis calls stay stubbed, as
they are on real DGX/HGX nodes.
nvlink:
switches:
- bdf: "0000:05:00.0"
uuid: "NVSwitch-H100-0000-0000-0000-000000000000"
device_id: 0x22A310DE # 10de:22a3 GH100 [H100 NVSwitch]
device_id is the packed (device << 16) | vendor word, the same form
device_defaults.pci.device_id takes for a GPU. It is optional, and it decides
whether the node can see the switch over PCIe at all:
device_id |
Meaning | Profiles |
|---|---|---|
| set | Baseboard switch on the node's PCIe bus. Rendered into the PCI tree as a bridge, so lspci lists it beside the GPUs — see NVSwitches on the PCI bus. |
a100, h100 |
| unset | NVLink endpoint only. The node never enumerates it, which is how a rack-scale switch tray presents. | gb200, gb300 |
NVLink error injection (per device)¶
A device can be given a rising NVLink DL error rate on its switch links, so its
uplinks report climbing replay/recovery/CRC errors — the GPU-side signal DCGM's
DCGM_HEALTH_WATCH_NVLINK reads (→ DCGM_FR_NVLINK_*). Set it in a device
override, or at runtime with nvml-mock-ctl nvlink-error:
devices:
- index: 0
nvlink_error:
rate: 250 # errors/second added on top of the healthy baseline
links: [0, 1, 2] # optional; omit to inject on all active links
The count accrues monotonically off the shared counter epoch (same model as
nvlink.defaults.error_rate); rate: 0 (or omitting the block) is healthy. This
is deliberately not an "NVSwitch entity health" knob — DCGM's NVSwitch/SXID
health is NSCQ/kernel-log sourced and cannot be driven from a libnvidia-ml.so
mock.
Fabric Health¶
fabric: describes the GPU's NVLink fabric attachment — the identity fields
(cluster_uuid, clique_id, state) plus the health nvidia-smi -q renders
under Fabric → Health. A fabric: block with no health keys means a healthy
fabric, so the shipped Grace-Blackwell profiles report Summary: Healthy and
Bandwidth: Full without configuring anything:
device_defaults:
fabric:
cluster_uuid: "00000000-0000-0000-0000-000000000001"
clique_id: 0
state: "auto" # couple registration to fake fabricmanager readiness
Fault a single condition by naming it. Every other condition stays healthy, so a consumer sees exactly the fault you configured:
health: key |
nvidia-smi -q row |
values |
|---|---|---|
degraded_bandwidth |
Bandwidth |
false → Full, true → Degraded |
route_recovery |
Route Recovery in progress |
false / true |
route_unhealthy |
Route Unhealthy |
false / true |
access_timeout_recovery |
Access Timeout Recovery |
false / true |
incorrect_configuration |
Incorrect Configuration |
none (default), no_partition, insufficient_nvlinks, incompatible_gpu_fw, invalid_location, incorrect_sysguid, incorrect_chassis_sn, gpu_state_invalid |
Health summary¶
The Summary row is derived from the conditions above, so an injected fault
moves it: all clear → Healthy, degraded bandwidth alone → Limited Capacity,
any other fault → Unhealthy. Pin it with health_summary when you need a
summary that disagrees with the conditions (a driver that reports a fault
without classifying it, say):
fabric:
health_summary: "limited_capacity" # healthy | unhealthy | limited_capacity
# | not_supported | auto (default)
not_supported reproduces the pre-#677 rendering: nvidia-smi treats an
unreported summary as "no health data" and prints N/A for the whole Health
block.
Raw health_mask¶
health_mask sets the NVML v2/v3 health bitmask directly, for encodings the
health: keys cannot express. It replaces the derived mask wholesale, and the
summary is derived from it:
An explicit health_mask: 0 means "the driver reported no health at all" and
renders the whole block as N/A — that is why the shipped profiles no longer
set it.
Partition Assigned¶
The mock reports this field as NOT_SUPPORTED, which is what a real GB300 tray
in a healthy rack reports (the driver does not answer it). Whether the row
appears at all is up to the nvidia-smi build: 580.173.02 prints
Partition Assigned : N/A, while the 580.65.06 binary the mock image bundles has
no such label and omits the row for every mask value.
Fabric health can also be degraded and restored at runtime with
nvml-mock-ctl fabric-health,
without restarting the consumer.
Available GPU Profiles¶
Standalone configuration files are provided for each supported GPU model:
| File | GPU Model | Memory | Architecture |
|---|---|---|---|
pkg/gpu/mocknvml/configs/mock-nvml-config-gb300.yaml |
NVIDIA GB300 NVL | 278 GiB | Blackwell Ultra |
pkg/gpu/mocknvml/configs/mock-nvml-config-gb200.yaml |
NVIDIA GB200 NVL | 186 GiB | Blackwell |
pkg/gpu/mocknvml/configs/mock-nvml-config-b200.yaml |
NVIDIA B200 | 180 GiB | Blackwell |
pkg/gpu/mocknvml/configs/mock-nvml-config-h100.yaml |
NVIDIA H100 80GB HBM3 | 80 GiB | Hopper |
pkg/gpu/mocknvml/configs/mock-nvml-config-a100.yaml |
NVIDIA A100-SXM4-40GB | 40 GiB | Ampere |
pkg/gpu/mocknvml/configs/mock-nvml-config-l40s.yaml |
NVIDIA L40S | 48 GiB | Ada Lovelace |
pkg/gpu/mocknvml/configs/mock-nvml-config-t4.yaml |
NVIDIA T4 | 16 GiB | Turing |
Each file contains a complete configuration with every device configured: 8 for
the baseboard profiles, 4 for the Grace-Blackwell ones (gb200, gb300), which
model one NVL72 compute tray, the unit a real node reports.
Integration Values¶
When deploying via Helm, additional values control integration with external projects:
| Key | Default | Description |
|---|---|---|
integrations.fakeGpuOperator.enabled |
false |
Create per-profile ConfigMaps named gpu-profile-<profile>, keyed profile.yaml, in the shape fake-gpu-operator's loader reads |
integrations.fakeGpuOperator.targetNamespace |
"" (release namespace) |
Namespace for the profile ConfigMaps. Set to FGO's release namespace for FGO to find them; requires FGO's builtinProfiles.enabled=false |
integrations.fakeGpuOperator.profileLabels |
run.ai/gpu-profile: "true" |
Extra labels on profile ConfigMaps. The contract labels are always emitted |
See the fake-gpu-operator guide for setup details.
Metric Fidelity¶
Every value the mock reports comes from configuration, from a clock-driven simulator, or from a fixed derivation — never from real GPU work, because the mock never executes a kernel. This section states which is which, so a dashboard built against nvml-mock is read with the right expectations.
Simulated — changes between calls¶
The driver is elapsed time, not workload: these numbers move on a completely idle node, and they do not move any faster under load.
Utilization is live in every shipped profile. Each profile carries its own
device_defaults.dynamic_metrics.utilization block — pattern: steady, GPU
10–45%, memory 5–25% — so nvmlDeviceGetUtilizationRates, and therefore
DCGM_FI_DEV_GPU_UTIL, never reports a constant 0 on a default install. The
floor is deliberately above 0 so "utilization is reported" is a deterministic
assertion rather than a flaky one. Because every DCGM_FI_PROF_* activity
metric is a fixed fraction of utilization.gpu (see Deliberately fixed
below), the whole profiling surface is non-zero as a consequence.
Temperature and power stay opt-in: set gpu.dynamicMetrics.enabled=true
(Helm) or add dynamic_metrics.temperature / dynamic_metrics.power to the
config. Note that enabling the Helm overlay replaces the profile's
dynamic_metrics block wholesale with the chart baseline, whose utilization
default is pattern: burst across 0–100 — that band's idle phase does report
near 0. Pin gpu.dynamicMetrics.utilization.* if you need a floor.
| Metric | Config | Behaviour |
|---|---|---|
temperature.gpu |
dynamic_metrics.temperature |
base_c, plus a sine ramp over ramp_period_sec bounded by ramp_c, plus uniform noise in ±variance_c; clamped to the thermal shutdown threshold when known |
power.draw |
dynamic_metrics.power |
base_mw ± variance_mw, clamped to min_limit_mw / max_limit_mw when those are set |
utilization.gpu, utilization.memory |
dynamic_metrics.utilization |
random within the configured band, shaped by pattern (idle / busy / burst / steady) |
| NVLink throughput counters | — | accrue deterministically from a process-independent epoch (MOCK_NVML_EPOCH, else /proc/stat btime), so they grow monotonically across separate nvidia-smi runs |
Static — held until the config changes¶
Everything else resolved through the effective config: device memory
(memory.total_bytes / free_bytes / used_bytes / reserved_bytes), ECC mode
and counters, clocks, fan speed, performance state, power limits, processes,
and failure injection. Each holds its configured value until the profile changes
or nvml-mock-ctl writes a runtime override, which takes effect within one
override TTL — see nvml-mock-ctl.
Three of these also accept writes from the consumer: the power management
limit, persistence mode and clocks. Clock tuning (nvidia-smi -ac, -lgc,
-lmc) works like the power limit below; see
Tuning clocks at runtime.
The power management limit (nvidia-smi -pl, in milliwatts and inclusive of
min_limit_mw / max_limit_mw; a cap outside those bounds is refused) is
recorded in the runtime override document, so it behaves like the driver-level
write it models: every process on the node reads the new cap, including ones
started afterwards, and it holds until a reset clears it (nvidia-smi -r, or
nvml-mock-ctl reset). A cap moves both the power management limit and the
enforced limit, but not default_limit_mw. A mock with nowhere to record the
write — no config, or an overrides file it cannot write — refuses the cap with
NVML_ERROR_NO_PERMISSION rather than reporting a success that nothing would
observe.
Persistence mode (nvidia-smi -pm) is recorded the same way, under the same
reset and the same NVML_ERROR_NO_PERMISSION refusal. It matters beyond its own
query: a GPU in persistence mode refuses to drain — see
Drain, removal and exclusion.
Workload power profiles¶
power.workload_power_profiles opts a device into the Blackwell workload power
profile feature, which nvidia-smi power-profiles reads and writes. Absent — the
default — the device declines the feature the way every pre-Blackwell board does.
requested seeds the set a consumer sees until one of the setters overrides it:
power:
workload_power_profiles:
supported:
- id: 0 # NVML_POWER_PROFILE_MAX_P, rendered "Max-P"
priority: 10 # lower value wins arbitration
conflicts: [1, 5] # cannot be enforced alongside these
- id: 6 # "LLM Inference"
priority: 40
requested: [] # profile ids to ask for
id is an NVML_POWER_PROFILE_* index (0-254) and doubles as the profile's bit
position in NVML's 255-bit masks, so ids must be unique. It is also what
nvidia-smi renders as the name: -l lists the supported set, -ld adds the
priority and conflicts, -gr and -ge report the requested and enforced sets,
and -sr / -cr add to and remove from the requested set.
Requested and enforced differ because asking for mutually exclusive profiles is
allowed: enforced is what survives arbitration, dropping any profile that
conflicts with a higher-priority one that was also requested. A requested id the
device does not advertise is ignored in config, but refused with
NVML_ERROR_INVALID_ARGUMENT when a consumer asks for it at runtime.
requested is empty in the shipped profiles, because every real GB200, GB300 and
B200 capture reports no requested or enforced profile — which is what
nvidia-smi -q -x renders as N/A in its <power_profiles> block.
A write outranks requested from the moment it lands, and like nvidia-smi -pl
it is recorded in the runtime override document rather than in the writing
process — so a later nvidia-smi reads it back, and it holds until a reset:
$ nvidia-smi power-profiles -sr 2 -i 0
Successfully set the requested profiles.
$ nvidia-smi power-profiles -gr -i 0
2. Compute
Within a single invocation only -ge reflects a write, because nvidia-smi
evaluates -sr and -cr before -ge but after -gr:
$ nvidia-smi power-profiles -sr 0,2 -cr 0 -ge -i 0
Successfully set the requested profiles.
Successfully cleared the requested profiles.
2. Compute
Because the request persists, a consumer that wants the board back at its configured state has to clear what it asked for, or reset the device. Two further nvidia-smi behaviours are worth knowing, neither of them the mock's: it refuses an id the board does not advertise before calling NVML at all, and it applies a comma-separated list in full only to the first GPU it visits, passing just the first profile to the rest.
Two axes decide whether the feature answers at all, and they fail differently. A
device that declares no workload_power_profiles reports the feature as
unsupported; one whose system.driver_version is older than 570 does not export
the symbols, so nvidia-smi cannot find the function. Of the shipped profiles only
gb200 and gb300 satisfy both; the others run a new enough driver but declare
no profiles.
Power smoothing¶
Power smoothing is the Blackwell feature that holds a GPU's power draw above a
floor so that load swings do not ripple back into the facility's supply. It
renders as the "Power Smoothing" block of nvidia-smi -q and is driven by
nvidia-smi power-smoothing. Blackwell and later devices support it by default,
which on the shipped profiles means b200, gb200 and gb300; earlier
architectures report it as N/A. It also needs a system.driver_version of 570
or later: an older driver reports none of the power smoothing fields and lacks the
setters. power.smoothing seeds its state:
power:
smoothing:
supported: true # override the architecture default
state: "enabled" # default "disabled"
active_profile: 1 # preset 0-4, default 0
profiles:
1:
percent_tmp_floor: 65.5 # floor as % of the power ceiling, 0-100
ramp_up_rate_mw_per_s: 30000 # rendered as 30.00 W/s
ramp_down_rate_mw_per_s: 15000
ramp_down_hysteresis_ms: 250
The device has five preset profiles, numbered 0 to 4. A preset the config does
not list reads as all zeros. The power ceiling ("Current TMP") is the enforced
power limit, so it follows nvidia-smi -pl. While smoothing is enabled, the floor
is that ceiling times the active preset's percent_tmp_floor. Disabled, the floor
is 0 W. Smoothing does not change the simulated power draw.
-s, -spp, -ptf, -rur, -rdr and -rdh are recorded in the runtime
override document like nvidia-smi -pl, so a later nvidia-smi reads them back
and they hold until a reset:
$ nvidia-smi power-smoothing -i 0 -s 1
Successfully Enabled Power Smoothing
$ nvidia-smi power-smoothing -i 0 -p 2 -rur 30000
Successfully updated Ramp Up Rate for Preset Profile ID 2
$ nvidia-smi power-smoothing -i 0 -spp 2
Successfully Activated Preset Profile ID: 2
$ nvidia-smi -i 0 -q | grep -A3 "Current Profile"
Current Profile
% TMP Floor : 0.00%
Ramp Up Rate : 30.00 W/s
Ramp Down Rate : 0.00 W/s
A preset outside 0-4, a floor outside 0-100 %, a negative ramp rate or a
fractional hysteresis is refused with NVML_ERROR_INVALID_ARGUMENT. Admin
overrides are not modelled and read Not Set; the privilege level reads N/A,
immediate ramp-down Disabled, and the remaining hardware lifetime 100 %.
Note
Captures from real B200, GB200 and GB300 nodes render the whole block as
Insufficient Permissions. The mock does not model NVML privileges, so it
always renders the full block.
Like workload power profiles, the feature also needs system.driver_version 570
or later; an older driver does not export the symbols.
PowerMizer mode and power source¶
power.power_mizer_mode sets the mode nvmlDeviceGetPowerMizerMode_v1 reports:
adaptive, prefer_maximum_performance, auto (the default) or
prefer_consistent_performance. Every mode is advertised as supported on
Maxwell and later; earlier architectures decline. nvmlDeviceSetPowerMizerMode_v1
writes the mode to the runtime override document, so other processes read it back.
Both entry points need system.driver_version 580 or later. nvidia-smi 580 does
not show this mode, so only an NVML consumer observes it.
nvmlDeviceGetPowerSource reports AC power on every device.
Deliberately fixed¶
Profiling metrics (DCGM_FI_PROF_*). DCGM reads these on Hopper+ through the
NVML GPM API. The mock derives every activity metric as a fixed fraction of
utilization.gpu:
| GPM metric | Fraction of utilization.gpu |
|---|---|
| SM utilization, graphics utilization | 1.00 |
| SM occupancy | 0.75 |
| Any-tensor activity | 0.50 |
| HMMA (FP16 tensor) | 0.40 |
| FP16 | 0.35 |
| FP32 | 0.25 |
| Integer | 0.10 |
| FP64, DMMA, IMMA | 0.05 |
| DFMA | 0.02 |
DRAM bandwidth activity mirrors utilization.memory, and PCIe / NVLink
throughput come from the counter snapshots. The fractions approximate the shape
of a tensor-dominated Hopper training step; they are intentionally not tied to
execution. SM occupancy and tensor-core activity are properties of kernels the
mock does not run, so deriving them from anything other than the configured
utilization would be fabrication rather than simulation. They stay fixed by
design and are not a gap to be closed.
With MIG enabled, a GPU instance can be sampled by its ID
(nvmlGpmMigSampleGet) or through one of its MIG device handles
(nvmlGpmSampleGet); both answer the same. The mock spreads the board's load
evenly across its slices, so every GPU instance reports the board's activity
percentages, while PCIe and NVLink throughput are its slice share of the
board's: a 1g instance on a seven-slice board reports one seventh of the
board's PCIe rate.
Conf Compute protected memory. The Conf Compute Protected Memory Usage
block of nvidia-smi -q reports 0 MiB for Total, Used and Free on every
profile, and there is no key to change it. Nothing partitions protected memory
until Confidential Compute mode is switched on, which the mock does not model
(see issue #377), and NVML reports 0 MiB rather than an unsupported query
whether or not the part could do CC: every board in
tests/e2e/go/assertions/nvidiasmi/testdata/hardware reports it that way,
including A100, L40S and T4. A profile pinned to a driver older than 525 is the
one exception — the CC APIs did not exist yet, so the rows read N/A, as they do
on real hardware of that vintage.
Board diagnostics. No key changes these answers. Wherever nvidia-smi -q
renders one, it matches the hardware captures of a freshly loaded driver:
| Query | Answer |
|---|---|
| Bus type | PCIe, on every profile including Grace boards |
EGM (capabilities block) |
disabled |
| PDI | A fixed 64-bit value derived from the GPU's UUID, the same in every process |
| Clock monitor | No faults, on Ampere and newer |
| InfoROM BBX flush | N/A: not flushed yet during this driver run |
| Bridge chips, IRQ number, PRM registers | Not supported |
Sample buffers. nvmlDeviceGetSamples reports 20 samples one second
apart, and every one of them carries the current reading of the matching
getter: power draw, GPU and memory utilization, the graphics and memory clocks,
and encoder, decoder, JPEG and OFA utilization. The mock keeps no history, so
the minimum, maximum and average over the window all equal the current value,
even under dynamic metrics. Timestamps
sit on a fixed one-second grid, so a poller that passes back the last
timestamp it saw gets only the samples taken since, and NVML_ERROR_NOT_FOUND
when there are none. On a GPU with MIG enabled, the GPU, memory, encoder and
decoder utilization samples are not supported. Module power samples are not
supported on any profile.
Identity and topology. Device name, architecture, brand,
compute_capability, uuid, PCI bus_id, and the BAR1 aperture
(bar1_memory) are baked onto the device at construction and need a pod restart
to change.
Allocation-aware — opt in¶
memory.used_bytes and memory.free_bytes follow Kubernetes GPU allocation
when allocationWatcher.enabled=true. A sidecar in the nvml-mock DaemonSet
polls the kubelet pod-resources API and writes the same runtime override file
nvml-mock-ctl writes, which the engine re-reads within one override TTL.
Scheduling a pod that claims a GPU moves that GPU's number; deleting the pod
returns it.
| Value | Behaviour |
|---|---|
allocationWatcher.usedFractionPerClaim |
Share of the usable aperture (total - reserved) attributed per claim. Default 0.5 |
| Time-sliced GPUs | Each holding container is a separate claim; claims on one device add up and clamp at the aperture |
| Unclaimed GPUs | Report used_bytes: 0 and the profile's idle free_bytes |
The reported bytes are synthetic. The mock runs no kernel and allocates no device memory, so the number says a claim exists — not what a workload touched. It reads per device: a claim on GPU 3 moves GPU 3 only.
The design is level-triggered. Each poll recomputes every device from the full current allocation, so a missed event self-heals on the next tick rather than leaving a GPU pinned at a stale value. A failed poll publishes nothing and keeps the previous reading, because publishing zeros on a transient kubelet error would report the whole node idle.
While the watcher runs it owns those two fields: an nvml-mock-ctl set --gpu
N memory.used_bytes=… is overwritten on the next poll. Every other field
nvml-mock-ctl writes is preserved — both writers take the same lock.
Off by default, because enabling it mounts the kubelet pod-resources socket (read-only, node-local; no RBAC, since it is not the API server).
Still not workload-aware — known gap¶
processes stays empty regardless of allocation, so nvidia-smi reports no
running processes even on a claimed GPU. Tracked in
#506 item 2.
Read the Simulated section carefully here too: utilization.gpu is no longer
a constant 0, but it is not evidence of work either. It moves on elapsed
time. A busy-looking utilization on an idle node is exactly what that block
produces, by design.
What each container can see has always been allocation-aware, and is a
separate thing from what the values say: the engine filters its visible GPU set
by which /dev/nvidia* nodes are present, so a pod allocated one GPU on an
eight-GPU node sees exactly one in nvidia-smi -L.
To move any of these explicitly, use nvml-mock-ctl set.
Validation¶
The configuration is validated on load:
versionfield is requiredsystem.driver_versionis required- Device indices must be unique
- Invalid YAML syntax causes fallback to defaults
Enable debug mode to see validation errors: