Libraries and Shims

An unmodified NVIDIA tool has to be made to see hardware that is not there, and there are only two ways to do it: replace the library it loads, or intercept the files it reads. Mokka does both, because it can own one and not the other.

A tool asks for	Mokka answers with	How
NVML — nvidia-smi, DCGM, the device plugin, the DRA driver	the mock libnvidia-ml.so	replacement
PCI sysfs — lspci, topology-aware schedulers	libmockfs.so	interception
InfiniBand — ibstat, ibv_devinfo, iblinkinfo	the libibmock* libraries	interception
the nvidia-imex binary	nvidia-imex-shim	exec wrapper

Why two mechanisms

NVML is a library, and a library can simply be replaced: Mokka ships a real libnvidia-ml.so at the path consumers already load, and owns the whole contract. Nothing is forwarded anywhere, because there is nothing behind it.

/sys and /dev are not Mokka's to own — they are the kernel's. A fake device tree can be staged elsewhere on disk, but lspci will still look at /sys/bus/pci. So those tools get their lookups rewritten instead, and read the staged tree at the paths they already use.

The practical consequence when debugging: wrong nvidia-smi output points at the library; wrong lspci or ibstat output points at a shim.

For where all of this sits in the wider system, see the architecture overview.

The mock NVML library

A real libnvidia-ml.so that consumers dlopen exactly as they would the vendor library — but it reads a YAML profile instead of a driver and a GPU.

It runs inside each consumer, not as a service

There is no daemon. Every process that loads the library gets its own copy of the engine, its own device objects and its own view of the node. Two consumers on the same node agree because they read the same configuration, not because they talk to anything shared.

That single fact explains most of the component's behaviour — including why runtime changes are delivered as a file rather than a command.

How a call is answered

flowchart LR
    c[C caller] -->|C symbol| bridge[CGo bridge]
    bridge -->|handle| engine[Engine]
    engine -->|lookup| device[ConfigurableDevice]
    yaml[(profile YAML)] --> engine

A consumer calls a C symbol. The bridge converts arguments across the CGo boundary and resolves the opaque device handle to a Go object. That object answers from configuration, and the value travels back out the same way, converted to C types at the bridge.

Handles are opaque pointers, valid from nvmlInit until the matching nvmlShutdown. Initialisation is reference-counted, so a consumer that initialises twice must shut down twice. On the final shutdown handles are invalidated and their addresses are never reused — a stale handle is rejected rather than silently resolving to a different device.

The library is safe to use from multiple threads.

What decides the answer

Two inputs, merged in a fixed order:

Input	Changes	Scope
The GPU profile	At install or helm upgrade	The node's static identity: model, count, memory, topology
Runtime overrides	At any time, without a restart	Health and telemetry: temperature, power, utilisation, clocks, ECC state, failure modes

Because the library is per-process, an override cannot be pushed to it. It is written to a file beside the profile instead, and every loaded copy re-reads that file on a short TTL and merges it over the base. Running and newly started processes converge within one interval, and the profile itself is never mutated. See Runtime Control.

Implemented versus stubbed

NVML is large, and not every function has a hand-written implementation. Those that do not still export a symbol and return NVML_ERROR_NOT_SUPPORTED.

This matters more than it sounds. A consumer that probes for a function gets a truthful "not supported" and takes its fallback path — the same thing it would do against an older real driver. A missing symbol would instead fail the dynamic link and crash the consumer at load.

For the current split:

go run ./cmd/generate-bridge -input $GO_NVML_DIR/pkg/nvml/nvml.go --stats

The stub layer is generated from NVML's own header, so adding a hand-written implementation removes its stub automatically. See the contributing guide for that workflow.

Limits

The library answers management calls. It has no data path: nothing computes, and no memory is allocated on a device that does not exist. Anything that needs a real GPU to do work rather than report state is out of scope.

CUDA simulation on CPU is not supported as of now. For the other surfaces Mokka does not simulate, see what is not simulated yet.

The shims

When a shim works, and when it does not

Two shims are LD_PRELOAD libraries. They interpose on the libc wrappers a program calls to touch the filesystem — open, stat, readlink, opendir and friends — and splice a prefix onto any path that matches.

That only works for programs that go through libc.

Go binaries bypass LD_PRELOAD entirely

Go makes syscalls directly rather than through libc, so a preloaded library is never consulted. Most of the Kubernetes control plane is Go, which is why Mokka also bind-mounts the staged tree at the real paths. The shims serve the C tools; the mounts serve everything else.

Both libraries are no-ops unless their environment variable is set, so a process that inherits them without the corresponding configuration sees the real host.

Shim	Kind	Makes this work
libmockfs	LD_PRELOAD	lspci and topology-aware schedulers see mock GPU BDFs
libibmock	LD_PRELOAD	ibstat, ibstatus, iblinkinfo, ibv_devinfo see mock InfiniBand HCAs
nvidia-imex-shim	execve wrapper	nvidia-imex starts on a machine with no GPU

libmockfs

Builds libmockfs.so. Redirects lookups under /sys/bus/pci, /sys/bus/pci/devices and /sys/devices/pci into the tree named by MOCK_PCI_ROOT, and is a no-op when that variable is unset.

libibmock

Builds three libraries, which is why the InfiniBand mock can be turned on in degrees rather than all at once:

Library	Covers
libibmocksys.so	sysfs and device-node path redirection
libibmockumad.so	the UMAD management interface
libibmockverbs.so	libibverbs

MOCK_IB selects how much of the fabric is simulated. The chart exposes it as infiniband.mockTier:

MOCK_IB	Effect
full	Path redirection plus UMAD and verbs, backed by the mock-ib daemon
sysfs	Path redirection only — tools enumerate devices but cannot open a verbs context
off, unset, or unrecognised	Every shim becomes a true no-op and the process sees the real host

Matching is case-insensitive, and there is no separate disable flag: leaving MOCK_IB unset is how you turn the shims off. Redirection targets MOCK_IB_ROOT, which the chart points at /var/lib/nvml-mock/ib.

The libraries target glibc 2.36 and later. Older __xstat-family symbols are also intercepted so the shim still behaves on earlier libc versions.

nvidia-imex-shim

Not an LD_PRELOAD library — a small Go program installed at /usr/bin/nvidia-imex, with the real daemon moved to /usr/bin/nvidia-imex.real.

The upstream compute-domain daemon hard-codes its command line with no flag passthrough, so --nogpu cannot be injected from outside. The shim appends it and execs the real binary, preserving arguments, environment and stdio. Because it execs rather than forks, no wrapper process lingers and signals reach the daemon directly.

This one is meant to disappear: it exists only until upstream supports passing extra arguments (#304).

How they reach a process

The chart preloads all four libraries on the node daemon, most specific first:

LD_PRELOAD=/usr/local/lib/libibmockumad.so.1:/usr/local/lib/libibmockverbs.so.1:\
/usr/local/lib/libibmocksys.so.1:/usr/local/lib/libmockfs.so.1

Workload containers receive them through the same CDI injection that delivers the rest of the mock, so a container that gets mock GPUs also gets the shims that make the C tools agree with them.

Building and testing

make mockfs-shim     # build libmockfs.so
make test-mockfs     # integration tests against real C test binaries

libmockfs ships test binaries under testbin/ that exercise the fortified open and fopen variants, because glibc's _FORTIFY_SOURCE rewrites those call sites and a shim that only hooks the plain symbols would silently miss them.

Related

To read about	See
How the whole system fits together	Architecture
What stages all of this onto a node	Node Daemon
Every profile knob	Configuration
Changing state at runtime	Runtime Control
Turning InfiniBand mocking on and off	Installation
Adding NVML functions or profiles	Contributing