Use in CI/CD¶
Run tests that need GPUs on ordinary CPU runners. Mokka installs into a throwaway Kubernetes cluster inside the job, your software talks to it as though the hardware were real, and the job tears everything down at the end.
This is the use case Mokka exists for: GPU runners are scarce and expensive, and most of what GPU software does in CI is discover and schedule hardware rather than compute on it.
The shape of a job¶
Four steps, in this order:
- Create a Kubernetes cluster on the runner — Kind is the usual choice.
- Install the Mokka chart: the latest release to start with, then pinned to the release your tests passed against.
- Wait for the DaemonSet to be ready.
- Run your tests against the cluster.
GitHub Actions¶
name: GPU tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Create a cluster
uses: helm/kind-action@v1
with:
cluster_name: mokka
- name: Install Mokka
run: |
helm install nvml-mock \
oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
--namespace mokka --create-namespace \
--set gpu.profile=gb300 \
--wait --timeout 5m
- name: Run tests
run: ./run-my-gpu-tests.sh
--wait is what makes this reliable: without it the next step starts before the
DaemonSet is serving, and your tests fail on a node that does not report GPUs
yet.
Pin the chart version¶
Without --version, helm install takes the latest stable Mokka release.
Once your tests pass, add --version with the release they passed against. An
unpinned chart means an upstream release can change what your CI sees on a day
you changed nothing, and the failure will not look like a version change.
Choosing a profile¶
The profile decides what hardware your software believes it found, so pick the one your tests assert against:
--set gpu.profile=a100 # 8 GPUs, 40 GiB each
--set gpu.profile=t4 # 4 GPUs, 16 GiB each — the smallest
--set gpu.profile=gb200 # 4 GPUs with NVLink fabric
To exercise more than one, run a job matrix over profiles rather than installing several releases into one cluster.
Testing failure handling¶
Mokka can present broken hardware, which is otherwise almost impossible to arrange in CI. A GPU that has fallen off the bus, an uncorrectable ECC error, or a device that reports as lost are all configuration:
To move a GPU between healthy and broken within a single job, use runtime control instead of reinstalling — it takes effect in about a second and needs no rollout.
What a plain cluster covers¶
A plain Kind cluster is enough for the mock driver and nvidia-smi, node
labels, the NVIDIA device plugin, the DRA driver, the GPU Operator and NRI
injection. Mokka's node daemon registers the nvidia runtime handler with
containerd and turns CDI and NRI on; see
Container runtime setup.
Keeping jobs fast¶
Most of the wall-clock is image pulls, not Mokka. Cache the runner's image layers
if your CI supports it, and prefer the smallest profile that still exercises what
you are testing — t4 presents four GPUs instead of eight.
Related¶
| To read about | See |
|---|---|
| Every chart value | Installation |
| What a profile defines | Configuration |
| Changing GPU state mid-job | Runtime Control |
| A failure-injection walkthrough | Failure injection |