Troubleshooting#

Common issues and their fixes. If you hit something not listed here, open an issue on the repository.

Setup-time issues#

Docker socket permission denied during model-server cleanup#

Symptom: launching model_servers prints a line like this for each port whose ownership cannot be inspected:

[<service>] cannot inspect :<port> ownership — not stopping

It can then print No persistent servers found running before failing with:

RuntimeError: could not stop persistent servers outside the profile

Running uv run model_servers --stop encounters the same inspection failure but ends with:

model-servers: failed to stop persistent servers: one or more persistent servers are still running

Cause: the login session lacks Docker socket access. This can affect DGX Spark and other Linux model-server hosts. Startup checks for persistent servers outside the selected profile before launching new ones. When Docker inspection fails, the empty-result message does not establish that no servers are running, and cleanup aborts because ownership could not be verified.

Diagnosis: run docker ps without sudo from the same login session. If it reports permission denied for /var/run/docker.sock, the launcher cannot inspect persistent containers. Its internal Docker check suppresses stderr, so running the command directly exposes the underlying error. These cleanup errors can have other causes; use the Docker output to confirm socket access is the problem.

Fix: complete the Docker group setup and refresh the launch session as described in Docker access and NVIDIA runtime. Verify that docker ps succeeds without sudo, then retry the model-server command.

DGX Spark — `uv sync` fails to build a wheel#

Symptom: uv sync fails on a DGX Spark system while building NeMo or vLLM wheels with errors mentioning missing Python.h or development headers.

Cause: the system is missing CPython development headers.

Fix: install before running uv sync:

sudo apt install python3-dev

This applies to the model-server-samples/model-servers/yaml/spark/ profile.

DGX Spark — CUDA allocation fails during a vLLM cold start#

Symptom: model_servers aborts while starting a vLLM service, often the embedding server after the larger services have started. The traceback reaches MemorySnapshot.measure(), then ends in torch.cuda.mem_get_info() or cudaMemGetInfo with a CUDA out-of-memory error. Another form ends in torch.AcceleratorError and references cudaErrorMemoryAllocation after the kernel driver reports NV_ERR_NO_MEMORY. Docker reports that the container was not OOM-killed, and htop can still show tens of gigabytes of available memory.

Cause: these tracebacks are not model-weight or KV-cache allocation failures. The CUDA driver fails to obtain immediately usable memory before or during initialization. The CUDA_LAUNCH_BLOCKING=1 line in the traceback is generic debugging advice, not the cause.

DGX Spark uses a unified memory architecture: the CPU, GPU, filesystem cache, and model servers share the same DRAM. Linux MemAvailable includes memory that the kernel can reclaim, while cudaMemGetInfo can report a smaller immediately free amount. NVIDIA documents this behavior in its DGX Spark unified-memory guidance, and a related vLLM issue describes how checkpoint reads held in the filesystem page cache can trigger a startup failure on UMA systems.

Diagnosis: inspect the host memory counters immediately after the failure:

grep -E '^(MemTotal|MemFree|MemAvailable|Cached|SReclaimable|SwapFree):' \
  /proc/meminfo

If MemAvailable is much larger than MemFree and Cached is large, the failure is consistent with reclaimable-cache pressure rather than the next model being too large for the configured profile.

Fix: the bundled spark profile enables spark_uma for each Docker-backed vLLM service. The wrapper downloads and syncs the complete Hugging Face snapshot before vLLM initializes CUDA, so transfer, reconstruction, and dirty writeback allocations do not overlap the CUDA context allocation. If the driver allocation still fails, the wrapper waits for reclamation and restarts the stopped container once. Only the wrapper that launched the current container attempt may restart it; another wrapper that adopts the running container observes it without taking ownership. The retry applies only when Docker reports that the container was not OOM-killed and the traceback reaches the initial vLLM memory snapshot through mem_get_info; model-weight and KV-cache OOMs are not retried.

If the retry is exhausted, first install the current DGX OS and driver updates. The DGX Spark release notes identify improved GB10 UMA out-of-memory handling in the July 2026 release. Stop unrelated memory-intensive workloads before retrying the stack.

For diagnosis and recovery, NVIDIA recommends flushing clean filesystem caches and then restarting the application:

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

The leading sync flushes dirty filesystem buffers. Dropping caches makes subsequent filesystem reads cold, so use this as a diagnostic workaround, not as routine launcher behavior.

Do not lower gpu_memory_utilization solely for this traceback. That setting cannot be evaluated when cudaMemGetInfo itself fails. Lower the setting only for later failures that report insufficient memory for the requested utilization or KV cache.

DGX Spark — vLLM reports insufficient KV cache only on a cold start#

Symptom: Nemotron Omni or the Cosmos VLM reports a negative or smaller KV-cache capacity on its first start than on a later start with the same configuration. The first start can reject the configured context even while Linux reports substantial available memory.

Cause: vLLM derives non-Torch usage from changes in globally available memory while it loads and profiles a model. On a unified-memory system, model downloads and checkpoint reads change the Linux filesystem page cache. vLLM can attribute part of that global change to the model and subtract it from the automatically sized KV cache. The behavior is tracked in vLLM issue #35920. The model-server launcher starts these services sequentially, so concurrent startup is not required to trigger the page-cache accounting error.

Fix: the bundled spark profile sets kv_cache_memory_bytes explicitly for both Nemotron Omni and Cosmos instead of using the fractional profiler to size their caches. Keep the fixed allocations when copying or modifying the profile. The values are 2 GiB for Omni’s 32,768-token hybrid Mamba/attention cache and 1.5 GiB for Cosmos’s 8,192-token cache. The Cosmos budget supports one maximum-length request while max_num_seqs: 4 retains concurrency for shorter requests. Concurrent requests near the context limit can queue, preempt, or recompute when their aggregate token demand exceeds the fixed cache. Increase the fixed cache when a custom Spark deployment needs parallel full-context requests.

The Spark files intentionally retain gpu_memory_utilization. In the bundled vLLM versions, kv_cache_memory_bytes controls the cache allocation and skips the unreliable profiling calculation, but vLLM still evaluates gpu_memory_utilization during its initial free-memory admission check. Do not remove the profile value and fall back to vLLM’s higher default.

Enable spark_uma on each Docker-backed vLLM service when copying the bundled settings into a custom Spark profile. Do not flush the filesystem cache routinely for this symptom: doing so makes the next checkpoint read cold and can reproduce the profiling variation. An explicit KV cache cannot bypass an earlier CUDA driver-allocation failure; the prefetch and bounded retry are separate safeguards for that stage.

DGX Spark — LOVR auto-download is not supported#

Symptom: uv run --project agent-samples/xr-render-demo xr_render_demo exits at startup with:

xr-render-demo: LOVR auto-download is not supported on linux/aarch64.

Cause: upstream LOVR releases do not ship a prebuilt aarch64 Linux binary, so the orchestrator cannot fetch one. Build LOVR from source on the Spark and point LOVR_BIN at it.

Fix:

sudo apt install -y cmake build-essential \
                    libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev \
                    libcurl4-openssl-dev libx11-xcb-dev

git clone --recursive https://github.com/bjornbytes/lovr.git ~/lovr
cd ~/lovr
mkdir build && cd build
cmake ..
make -j$(nproc)

export LOVR_BIN=~/lovr/build/bin/lovr

export LOVR_BIN=… only lasts for the current shell. To make it permanent, append the line to ~/.bashrc, or set lovr_bin: ~/lovr/build/bin/lovr in agent-samples/xr-render-demo/scene/scene_service.yaml instead.

If git clone was run without --recursive, run git submodule update --init --recursive inside ~/lovr before cmake ...

Blackwell GPUs (B200, RTX PRO 6000) — VLM fails to start#

Symptom: the VLM server logs FlashInfer or NVFP4 kernel errors and never becomes healthy on a Blackwell-class system.

Cause: The Docker backend cannot expose the Blackwell GPU, or the selected vLLM image does not contain compatible kernels. Any first-use kernel compilation occurs inside the container; host NVCC is not required.

Fix: follow Docker access and NVIDIA runtime to verify Docker access, register the NVIDIA runtime if needed, and check GPU access from a container. Retain the vLLM image pinned by the reviewed hardware profile.

This applies to the model-server-samples/model-servers/yaml/96G_blackwell/ profile.

GPU service aborts with `cuDNN version incompatibility`#

Symptom: a GPU service (commonly the NeMo STT server) crashes at torch import with:

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against
(9, 20, 0) but found runtime version (9, 13, 1). ... Looks like your
LD_LIBRARY_PATH contains incompatible version of cudnn.

Cause: the host exports an LD_LIBRARY_PATH that points at a system cuDNN (common on cloud GPU images). It shadows the cuDNN bundled in the service’s venv — the exact version that venv’s PyTorch was compiled against — so torch loads the wrong runtime and aborts.

Fix: the launcher handles this automatically — model_servers or xr_render_demo strip any libcudnn-bearing directory from each child’s LD_LIBRARY_PATH before spawning (logged once as a WARNING), so the venv-bundled cuDNN is used. If you hit this running a service directly (outside the launcher), clear the conflicting path yourself first:

# Inspect what's on the path
echo "$LD_LIBRARY_PATH"
# Run the service without the host cuDNN shadowing the venv copy
env -u LD_LIBRARY_PATH uv run <command>

`vllm_backend: docker` — image pull fails with “unauthorized” or “denied”#

Symptom: the wrapper logs [<service>] Launching vLLM (docker) and then docker run fails with one of:

Error response from daemon: pull access denied for nvcr.io/nvidia/vllm
unauthorized: authentication required
denied: requested access to the resource is denied

Cause: docker is not authenticated to nvcr.io, so it cannot pull the NGC vLLM container.

Fix: log in with your NGC API key once. Get a key from https://ngc.nvidia.com/setup/api-key and run:

docker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY

The credential is cached in ~/.docker/config.json and reused on subsequent runs. Alternatively, save the key into the xr-ai credential cache so the wrapper can auto-login:

python3 -c "
import json, os, pathlib
p = pathlib.Path.home() / '.config/xr-ai/credentials.json'
d = json.loads(p.read_text()) if p.exists() else {}
d['NGC_API_KEY'] = os.environ['NGC_API_KEY']
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(json.dumps(d, indent=2))
"

The orchestrator’s load_credentials() injects NGC_API_KEY into the environment before each wrapper runs; the docker backend uses it to run docker login nvcr.io --password-stdin when no existing auth is found.

`vllm_backend: docker` — wrapper exits with `vLLM exited before /health became reachable`#

Symptom: the launcher reports

[<service>] vLLM exited before /health became reachable
[<service>] exited (rc=1) before signaling ready

and the per-run log file under /tmp/log_<sample>_<timestamp>/<service>.log contains only wrapper messages — nothing from inside the container.

Health probe — confirm vLLM never reached the /health endpoint:

curl -fsS http://127.0.0.1:8108/health   # nemotron_omni (default LLM)
curl -fsS http://127.0.0.1:8100/health   # vlm_server (default Cosmos VLM)
curl -fsS http://127.0.0.1:8107/health   # superseded nemotron3_nano

Container post-mortem — during the current run, the wrapper streams docker logs -f into a sibling /tmp/log_<sample>_<timestamp>/xr-ai-vllm-*.log file. To inspect the retained container manually:

docker ps -a --filter name=xr-ai-vllm-
docker logs --tail=200 <container-name>

Cause: vLLM crashed during startup — common reasons: model weights missing or inaccessible in the bind-mounted model_cache, the nvidia runtime not exposing the selected GPU, HF token missing for a gated model, or a reasoning-parser plugin file that is not present inside the container.

Fix: read the current run’s container log, address the root cause shown there, and retry. Failed containers remain available to docker logs until managed cleanup removes them.

`vllm_backend: docker` — the NVIDIA runtime is unavailable#

Symptom: Docker is reachable, but container startup fails with:

docker: Error response from daemon: unknown or invalid runtime name: nvidia

Cause: the Docker daemon has no registered nvidia runtime. This can happen on DGX Spark even when the NVIDIA Container Toolkit is already installed. The model-server wrappers request --runtime=nvidia explicitly.

Fix: follow Docker access and NVIDIA runtime to install the toolkit if needed, register the runtime with nvidia-ctk, and restart Docker. Verify that docker info --format '{{json .Runtimes}}' lists nvidia, then run the GPU smoke test there before retrying the model servers.

If nvidia is listed but startup instead reports an nvidia-container-cli initialization error, inspect that error for a driver or GPU-access problem. Refer to the NVIDIA Container Toolkit troubleshooting guide.

Hub fails immediately because NVIDIA codec libraries are missing#

Cause: The hub raises RuntimeError: missing libnvcuvid.so or libnvidia-encode.so because NVDEC (libnvcuvid.so) and NVENC (libnvidia-encode.so) are required. The DeviceIOHub refuses to start without them so it never silently falls back to OpenH264, which is royalty-bearing.

Fix:

Bare metal: install or repair the NVIDIA driver. The libraries ship with the driver, not with CUDA.
Docker: use the NVIDIA runtime, request the needed devices with NVIDIA_VISIBLE_DEVICES, and include video in NVIDIA_DRIVER_CAPABILITIES when starting the container.

Runtime and connection issues#

Voice session drops or agent goes silent after a few minutes idle#

Cause: an idle-timeout that auto-cancels the voice pipeline after a stretch with no user or bot speech.

Status: disabled by default. VoiceAgent leaves its private Pipecat idle timeout disabled, so a quiet session stays connected indefinitely.

If you want it: set idle_timeout_secs: <seconds> (e.g. 300 for 5 min) in the sample’s worker YAML (simple_vlm_example_worker.yaml or xr_render_demo_worker.yaml); 0 or unset keeps it disabled. The knob is owned by xr_ai_voice.VoiceAgent.

Browser client connects but no audio or no video#

Most common cause: firewall blocking WebRTC media on UDP 7882 (LiveKit).

Fix: open ports per Networking and firewall. The web client will appear to connect (signaling through the port 8080 proxy succeeds) but media frames are silently dropped without 7882.

Audio-only cause: the browser’s autoplay policy blocked the remote audio track. Use the Connect button to open the session. If the session is already connected but silent, send a message or click Start Microphone to retry playback from a user gesture.

HTTPS web client → `ws://` mixed-content warning#

Symptom: the LiveKit JS SDK logs a mixed-content error connecting to ws://<host>:7880/… from an HTTPS page.

Cause: a stale client build (or a hand-rolled configuration) is pointing at LiveKit’s native 7880 instead of the same-origin wss://<host>:8080/rtc proxy the DeviceIOHub exposes.

Fix: rebuild against the current client-samples/web or web-xr — both auto-detect the page’s protocol and use the wss proxy. If you’re holding a LiveKitConfig directly, set port to the hub’s web_server_port (8080) and let the SDK build wss://host:8080. Android, iOS, and visionOS use wss only; there is no secure toggle.

Android — connection fails with TLS or certificate errors#

Symptom: the Android sample fails to connect; the error shows an SSLHandshakeException or similar TLS error.

Cause: the hub uses a development root CA by default. The Android sample validates the signed server leaf against the system and user CA store, the same as iOS.

Fix: install the hub’s certificate via the in-app button before connecting:

In the Connection section, tap Install hub certificate (enabled once Host is non-empty).
The app fetches the public root CA from https://<host>:<port>/cert and opens the system certificate-install dialog.
Confirm the install. After install, tap Connect — validation succeeds automatically.

Repeat for each hub host. Replace the auto-generated certificate with one from a public CA via cert_file and key_file in device_io_hub.yaml for production.

iOS and visionOS — connection fails with certificate-trust errors#

Symptom: the iOS or visionOS sample fails to connect; the LiveKit WebSocket reports a TLS error (e.g. NSURLErrorServerCertificateUntrusted, -1202, “The certificate for this server is invalid”).

Cause: the LiveKit Swift SDK cannot bypass certificate validation, so the hub’s development root CA must be trusted at the OS level.

Fix: install the hub’s certificate as a trusted profile on the device:

In Safari on the device, open https://<host>:8080/cert and tap Show Details → visit this website past the certificate warning.
Approve the Download Configuration Profile prompt.
Install via Settings → General → VPN & Device Management.
Toggle Settings → General → About → Certificate Trust Settings → Enable Full Trust for the new certificate.

If step 4 shows no toggle, remove the installed profile via VPN & Device Management, restart the hub, and install /cert again. Older xr-ai builds used web-server.crt itself as the trust profile. DeviceIOHub detects that legacy CA-as-server certificate and migrates it to root-ca.crt plus a signed CA:FALSE server leaf with a clear TLS: migrating legacy CA-as-server message. This migration requires one root reinstall; later server-leaf changes do not.

If the toggle was enabled but the wss handshake still fails with errSSLBadCert or NSURLErrorDomain -1202 and a message like “pretending to be 192.168.1.42” (that is, the IP you typed into the app), the server leaf’s SubjectAlternativeName doesn’t cover that IP. The hub detects local IPv4 addresses via a UDP-connect probe and auto-regenerates only the leaf whenever the SAN is missing one (logged as TLS: generated server leaf …); just restart the hub. The already installed root remains valid. If the dialed address is absent from the leaf, add it to web_server_extra_sans in device_io_hub.yaml and restart. Refer to Networking for details. To force leaf regeneration, delete ~/.local/share/xr-ai/web-server.crt and web-server.key before restarting. Do not delete root-ca.crt or root-ca.key unless you intend to reinstall the root on every client.

If the certificate is trusted (no -1202) but the room connection still fails with HTTP 401 or “no permissions to access the room”, the hub’s /rtc WSS proxy is dropping the Authorization: Bearer <token> header the Swift SDK sends. Update to the latest DeviceIOHub and restart; the proxy forwards the Authorization header on /rtc/validate and the WebSocket.

Repeat the install step per hub host, or replace the auto-generated certificate with a public-CA certificate via cert_file and key_file in device_io_hub.yaml for production.

iOS and visionOS — microphone or camera is interrupted#

An occasional LiveKit microphone timeout means its recording engine did not produce the first buffer before publication. The current client enables prepared recording mode before publishing to make that first buffer available. Stopping the microphone then disables prepared input while leaving output active, so the orange microphone indicator clears without silencing agent audio.

Phone calls, Siri, route changes, media-service resets, another capture app, or closing an XR space can interrupt audio or camera while the control still shows the user’s requested state. The client re-arms capture when the OS allows it. If it does not recover, filter Console.app for the MediaSession category to inspect the recorded interruption, route, and capture-session events. CoreAudio -50 and FigAudioSession -19224 messages alone are not evidence of failure; they can also appear on successful starts.

Chrome — Immersive Web extension cannot be enabled#

Symptom: the Immersive Web extension for Chrome cannot be enabled.

Status: known issue, no workaround currently.

Workaround: use a native client (Quest 3, Vision Pro) on the same LAN, or the IWER emulator built into the web client itself for desktop dev.

vLLM cold start takes 3–8 minutes#

Symptom: a vLLM server’s weight load is fast, but the server then sits silent for several minutes before becoming healthy.

Cause: CUDA graph capture and, for FP4 MoE models, FlashInfer autotuning happen on first run after weight load. They are silent.

Fix: the default Omni profiles set enforce_eager: false to enable CUDA graph capture and maximize steady-state throughput, so this startup delay is expected. For development, set enforce_eager: true in the active model YAML to skip CUDA graph capture. Eager mode starts faster but can reduce per-token throughput; keep the default when steady-state performance matters more than cold-start time.

vLLM exits before readiness with insufficient GPU memory#

Symptom: a vLLM container exits during startup with CUDA out of memory, No available memory for the cache blocks, or a negative available KV-cache value buried in its container log.

Fix: the wrapper classifies these signatures as INSUFFICIENT GPU MEMORY and prints the configured utilization when available. Free memory held by other processes, reduce model context or concurrency, or use a device with more GPU-visible memory. The complete original error remains in the reported log file.

`xr_render_demo` exits but VRAM is still pinned#

By design. The vLLM-backed servers (nemotron_omni_llm_server, vlm_server, and nemotron3_nano_llm_server) survive stack restarts so model weights stay loaded across worker crashes and debug restarts. Refer to AI inference servers → vLLM model persistence.

Fix: to fully release VRAM:

cd xr-ai
uv run --project model-server-samples/model-servers model_servers --stop

For Pocket TTS this sends SIGTERM to its verified launcher-owned process group, waits up to 20 s, then sends SIGKILL. Pocket TTS remains in that group so the same escalation also reaches it during an aborted stack startup. Cleanup falls back to the listener PID if process-group ownership cannot be verified, and other local servers always use PID-only cleanup. For docker-mode servers it runs docker stop <container_name> (escalating to docker kill after 20 s). Safe to run while the stack is down. The command exits nonzero rather than reporting success if server ownership cannot be verified or any target remains running.

First run downloads models silently#

Symptom: uv run --project model-server-samples/model-servers model_servers appears to hang at startup the first time.

Cause: model weights are downloading from HuggingFace into models/ at the repository root (gitignored; the Cosmos3 checkpoint alone is tens of GB).

Fix: wait. Subsequent runs use the cached weights and start in ~30–60 s. If the download makes no progress, note that unauthenticated downloads (model-server runs started with --allow-anonymous) are rate-limited and can stall indefinitely; set HF_TOKEN and restart. Refer to Credentials.