Troubleshooting Guide

Common issues and solutions for Mock NVML.

Build Issues

CGo Not Enabled

Error:

cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

Solution:

# Ubuntu/Debian
sudo apt-get install build-essential

# RHEL/CentOS
sudo yum groupinstall "Development Tools"

# macOS
xcode-select --install

Go Version Too Old

Error:

go: go.mod file indicates go 1.25

Solution:

# Install Go 1.25+
wget https://go.dev/dl/go1.25.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.25.0.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Build on macOS

Problem: Cannot build shared library on macOS for Linux.

Solution: Use Docker build:

make docker-build

Runtime Issues

Library Not Found

Error:

nvidia-smi: error while loading shared libraries: libnvidia-ml.so.1: cannot open shared object file

Solution:

# Verify library exists
ls -la pkg/gpu/mocknvml/libnvidia-ml.so*

# Set library path correctly
export LD_LIBRARY_PATH=$(pwd)/pkg/gpu/mocknvml:$LD_LIBRARY_PATH

# Or use absolute path
LD_LIBRARY_PATH=/full/path/to/pkg/gpu/mocknvml nvidia-smi

Symbol Not Found

Error:

nvidia-smi: symbol lookup error: nvidia-smi: undefined symbol: nvmlSomeFunction

Solution:

1. Check if symbol is exported:

   nm -D pkg/gpu/mocknvml/libnvidia-ml.so | grep nvmlSomeFunction
   

2. If missing, check if it's a stub or needs implementation:

   # Check if it's in stubs
   grep "nvmlSomeFunction" pkg/gpu/mocknvml/bridge/stubs_generated.go
   
   # If stub, implement it in appropriate bridge file, then:
   go generate ./pkg/gpu/mocknvml/bridge/
   make -C pkg/gpu/mocknvml clean
   make -C pkg/gpu/mocknvml
   

Segmentation Fault

Error:

Segmentation fault (core dumped)

Causes and Solutions:

1. Handle dereferencing issue:
2. Enable debug mode: MOCK_NVML_DEBUG=1
3. Check which function crashes

4. Verify handle table is working

5. CGo memory issue:

6. Run with address sanitizer:

   CGO_CFLAGS="-fsanitize=address" go build -buildmode=c-shared ...
   

7. nvidia-smi version mismatch:

8. Ensure driver version in config matches nvidia-smi expectation
9. Try different nvidia-smi versions

YAML Config Not Loading

Error:

[CONFIG] Failed to load YAML config from /path/to/config.yaml: ...

Solution:

1. Verify file exists and is readable:

   cat $MOCK_NVML_CONFIG
   

2. Validate YAML syntax:

   python3 -c "import yaml; yaml.safe_load(open('$MOCK_NVML_CONFIG'))"
   

3. Check required fields:

   version: "1.0"                    # Required
   system:
     driver_version: "550.163.01"    # Required
   

4. Enable debug to see specific error:

   MOCK_NVML_DEBUG=1 LD_LIBRARY_PATH=. nvidia-smi
   

nvidia-smi Shows Wrong Values

Problem: Values don't match YAML config.

Solution:

1. Verify config is loaded:

   MOCK_NVML_DEBUG=1 LD_LIBRARY_PATH=. nvidia-smi 2>&1 | head -5
   # Should show: [CONFIG] Loaded YAML config: N devices, driver X.Y.Z
   

2. Check device override vs defaults:

3. Per-device settings override defaults

4. Check devices: section in YAML

5. Verify field mapping:

6. Some nvidia-smi fields map to different config fields
7. Check Configuration Reference

NOT_SUPPORTED Errors

Problem: nvidia-smi shows "N/A" or errors for some queries.

Explanation: Functions return NOT_SUPPORTED when: - YAML config doesn't provide the value - Function is not implemented (stub)

Solution:

1. Add missing config values:

   device_defaults:
     thermal:
       temperature_gpu_c: 33    # Add this for temperature
   

2. Check if function is implemented:

   # Check engine implementation
   grep "GetXxx" pkg/gpu/mocknvml/engine/device.go
   
   # Check bridge implementation
   grep "nvmlDeviceGetXxx" pkg/gpu/mocknvml/bridge/*.go
   
   # Check if it's just a stub
   grep "nvmlDeviceGetXxx" pkg/gpu/mocknvml/bridge/stubs_generated.go
   

Testing Issues

Integration Test Fails

Error:

Error: No GPU found

Solution:

1. Ensure mock library is built:

   make -C pkg/gpu/mocknvml
   

2. Check library is accessible in Docker:

   docker run -v $(pwd)/pkg/gpu/mocknvml:/mock -e LD_LIBRARY_PATH=/mock ...
   

Race Conditions

Error:

WARNING: DATA RACE

Solution: - All Engine methods should use mutex - Check that new code acquires locks properly - Run tests with race detector: go test -race ./...

Performance Issues

Slow Startup

Problem: nvidia-smi takes long to start.

Causes: 1. Large YAML config parsing 2. Debug logging enabled

Solution:

# Disable debug
unset MOCK_NVML_DEBUG

# Use simpler config for testing

Memory Usage

Problem: High memory usage.

Explanation: - Each handle allocates ~40 bytes of C memory - Error string cache grows with unique errors

This is typically not a problem for normal usage.

Environment Issues

LD_LIBRARY_PATH Not Working

Problem: System uses real NVML despite setting LD_LIBRARY_PATH.

Solutions:

1. Check library search order:

   LD_DEBUG=libs nvidia-smi 2>&1 | grep nvml
   

2. Verify no cached libraries:

   sudo ldconfig  # Refresh cache
   

3. Use absolute path:

   LD_LIBRARY_PATH=/absolute/path/to/mocknvml nvidia-smi
   

4. Check for ld.so.conf entries:

   grep nvml /etc/ld.so.conf.d/*
   

Docker Issues

Problem: Library doesn't work in Docker container.

Solutions:

1. Mount library correctly:

   docker run -v $(pwd)/pkg/gpu/mocknvml:/mock:ro \
              -e LD_LIBRARY_PATH=/mock \
              ubuntu nvidia-smi
   

2. Ensure nvidia-smi is available:

   docker run ... which nvidia-smi
   

3. Use correct architecture:

   # Build for container architecture
   GOARCH=amd64 make -C pkg/gpu/mocknvml
   

kind load docker-image Fails with "content digest not found"

Problem: Loading the published image into a kind cluster fails:

ERROR: command "docker exec --privileged -i <node> ctr --namespace=k8s.io images import -"
failed with error: exit status 1
ctr: content digest sha256:...: not found

Cause: ghcr.io/nvidia/nvml-mock is a multi-arch image. Docker Desktop's containerd image store keeps the whole manifest list, and kind load docker-image hands the node an archive whose per-platform layers it does not have. The error names the digest, not the cause. This is the default Docker Desktop configuration on macOS.

Solution: save a single platform and load the archive:

ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')
docker save --platform "linux/${ARCH}" ghcr.io/nvidia/nvml-mock:latest -o nvml-mock.tar
kind load image-archive nvml-mock.tar --name <cluster>

The platform must match the kind node's architecture. A locally built image (docker build -t nvml-mock:local ...) is single-arch already and loads with kind load docker-image unchanged.

GPU Operator Issues

Validator Pod in CrashLoopBackOff

Problem: The GPU Operator validator pod repeatedly crashes.

Explanation: The GPU Operator validator (/usr/bin/nvidia-validator) is a statically-linked Go binary with no shell. It probes the driver root for expected files.

Solution: Ensure the mock driver root has all required files:

# Use the specific Kind cluster and node name (adjust cluster name as needed)
NODE_CONTAINER="nvml-mock-operator-control-plane"
docker exec "$NODE_CONTAINER" ls -la /run/nvidia/driver/usr/lib64/libnvidia-ml.so*
docker exec "$NODE_CONTAINER" cat /var/lib/nvml-mock/driver/config/config.yaml

CDI Spec Not Generated

Problem: CDI specs are not appearing in /var/run/cdi/.

Solution: Check nvml-mock DaemonSet logs for CDI generation errors:

kubectl logs -l app.kubernetes.io/name=nvml-mock | grep -i cdi

Device Plugin Shows 0 GPUs with GPU Operator

Problem: The GPU Operator-managed device plugin reports 0 GPUs.

Solution: Verify the device plugin is using the mock driver root:

kubectl -n gpu-operator logs -l app=nvidia-device-plugin-daemonset | head -20

The GPU Operator must be installed with --set driver.enabled=false and --set toolkit.enabled=false when using mock GPUs, since there is no real NVIDIA driver to manage.

Getting Help

Debug Information to Collect

When reporting issues, include:

1. Environment:

   uname -a
   go version
   gcc --version
   

2. Debug output:

   MOCK_NVML_DEBUG=1 LD_LIBRARY_PATH=. nvidia-smi 2>&1
   

3. Library info:

   file pkg/gpu/mocknvml/libnvidia-ml.so
   ldd pkg/gpu/mocknvml/libnvidia-ml.so
   

4. Config file (if using YAML)

5. Expected vs actual output

Filing Issues

Include: - Steps to reproduce - Expected behavior - Actual behavior - Debug output - Environment info