AI inference servers#

Overview#

Build your application’s speech, vision, and reasoning capabilities using the shared inference services. Your workers call these services through model interfaces, allowing you to choose supported models and local or hosted endpoints as your application develops.

Available services#

Launch these inference and typed capability services alongside services/device-io-hub/. Model HTTP stays inside xr-ai-models; workers must not add vendor SDKs or hand-written HTTP clients.

Server

Command

Port

Model

Backend

services/vlm-server/

vlm_server

8100

Cosmos3 Nano Reasoner

vLLM (pip or docker)

services/stt-server/

stt_server

8103

parakeet-tdt-0.6b-v3

NeMo ASR in-process

services/speaker-stt/

speaker_stt

8102

Nemotron 3 Diarization + Multitalker Parakeet

NeMo streaming ASR

services/magpie-tts/

magpie_tts_server

8104

magpie_tts_multilingual_357m

NeMo TTS in-process

services/pocket-tts/

pocket_tts_server

8105

kyutai/pocket-tts

Pocket TTS in-process

services/nemotron3-nano-llm/

nemotron3_nano_llm_server

8107

NVIDIA-Nemotron-3-Nano-30B-A3B-{NVFP4,FP8}

vLLM (pip or docker)

services/nemotron-omni-llm/

nemotron_omni_llm_server

8108

Nemotron-3-Nano-Omni-30B-A3B-Reasoning (NVFP4, FP8, or BF16, GPU-selected)

vLLM (pip or docker) — multimodal (text + video)

services/clef-server/

clef_server

8120

Cloudflare Clef-Flash 9B

Native SystemOne choice decisions

services/embedding-server/

embedding_server

8109

Nemotron-3-Embed-1B-BF16

vLLM (pip or docker)

services/ocr-server/

Operator-run container

8000 (container); 8112 in example

Nemotron OCR v2 multilingual

Hugging Face in a local CUDA container

services/video-memory-service/

video_memory_service

8310

—

Typed recorded-video capability

services/rag-service/

rag_service

8340

—

Typed dense document retrieval capability

For model configuration and the launch workflow, refer to Customizing model servers. For the orchestrator pattern, refer to Adding a new sample.

Launcher-managed local model artifacts land in the service’s model_cache directory, set per YAML and resolved relative to the YAML file. Operators configure self-hosted NIM caches for their optimized engines and weights; hosted endpoints keep no weights in this repository. Every models/ tree is excluded from version control. The model-servers profiles share models/ at the repository root; the exact layout per launch style is below.

OCR serving#

The reusable services/ocr-server provides multilingual Nemotron OCR v2 in an operator-managed container. No supplied sample launches or consumes OCR. The Dockerfile builds an isolated Python 3.12, PyTorch 2.9.1, CUDA 13 runtime with NVIDIA’s pinned Hugging Face source. At startup, the service downloads v2_multilingual weights and keeps the model loaded across requests. Docker, NVIDIA Container Toolkit, and a CUDA 13-compatible NVIDIA driver are required; a host CUDA toolkit is not. The first build takes several minutes. Refer to the upstream installation instructions.

Set the build argument TORCH_CUDA_ARCH_LIST for the selected GPU: 8.9 for Ada, 12.0 for Blackwell, or 12.1 for Spark. From the repository root, build the image; this example targets Blackwell:

docker build --build-arg TORCH_CUDA_ARCH_LIST=12.0 \
  -t xr-ai-nemotron-ocr:local services/ocr-server/ocr_server

Run it in the foreground on GPU 0, publishing the API only on localhost:

mkdir -p models
docker run --rm --name xr-ai-ocr-manual --gpus device=0 --ipc=host \
  -e HF_TOKEN -e OCR_INFER_LENGTH=1024 \
  -p 127.0.0.1:8112:8000 -v "$PWD/models:/models" \
  xr-ai-nemotron-ocr:local

The operator owns GPU placement, container startup and shutdown, cache mounts, and readiness checks. After startup completes, check readiness in another terminal:

curl --fail http://localhost:8112/v1/health/ready

The service loads and warms the model before accepting requests. There is no host ocr_server command or run_model_stack registration. Sample stack stop commands do not stop this container, and no automatic reuse or replacement is performed. Provision GPU capacity before running it alongside other services; no GPU allocations are changed and coexistence is not qualified. Repository GPU tests do not discover arbitrary operator-run containers. Stop this container before running GPU tests that use the same device; the HTTP-only OCR smoke test below can run against it without managing containers.

For NIM, deploy Nemotron OCR v2 independently and configure its endpoint in the consumer profile. The operator selects the multilingual variant and owns GPU placement, model downloads, caches, and container lifecycle. XR AI does not start an OCR NIM container or require a local HTTP proxy. Refer to Deploying samples with NIM endpoints for the endpoint example and validation.

Both backends expose POST /v1/ocr and readiness at GET /v1/health/ready. Use the nemotron_ocr preset and typed SDK factory:

from pathlib import Path
from xr_ai_models import load_models_config_from_dict, make_ocr

config = load_models_config_from_dict({"ocr": {
    "adapter": {"preset": "nemotron_ocr"},
    "endpoint": {"base_url": "http://localhost:8112"},
    "deployment": {"ownership": "external"},
}})
async with make_ocr(config, "ocr") as ocr:
    regions = await ocr.read_text(Path("image.png"))
    for region in regions:
        print(region.text, region.confidence, region.polygon)

Applications reuse the endpoint; an OCR tool does not load its own GPU model. The client sends one inline PNG or JPEG and returns text, confidence, and normalized polygons. Remote image URLs are intentionally unsupported. Word, sentence, and paragraph grouping are available; recognition results need not be identical across backends.

Send an opaque image. The local Hugging Face server converts inputs to RGB without compositing transparency: opaque black text on a transparent black background becomes uniformly black, losing the visible text. Composite transparent PNGs onto an appropriate background before calling either backend. For dark text intended for a white background:

from PIL import Image

with Image.open("input.png") as source:
    rgba = source.convert("RGBA")
    opaque = Image.new("RGBA", rgba.size, "white")
    opaque.alpha_composite(rgba)
    opaque.convert("RGB").save("ocr-input.png")

Choose a contrasting background for light text. This preparation remains caller-owned; the service does not infer the intended background color.

The Hugging Face server clips finite bounding-box coordinates to the image’s normalized [0, 1] bounds while preserving recognized text and confidence. Non-finite model coordinates are an inference failure (HTTP 500), not valid detections. The SDK continues to reject malformed responses from any endpoint.

The Hugging Face server accepts one image per request, at most 10 MiB of PNG or JPEG file bytes after base64 decoding, and 20 million pixels after full-resolution square padding: max(width, height)² <= 20,000,000 (longest side at most 4,472 pixels). This check runs before RGB conversion or inference because upstream pads the image on CUDA before resizing; lowering infer_length does not reduce that allocation. The padded FP16 RGB tensor alone is bounded to about 114 MiB; this is not a bound on total inference memory. Requests that exceed the schema’s data-URL string-length limit receive HTTP 422 before image decoding. Requests that pass that limit but exceed the decoded image-file byte limit or padded pixel limit receive HTTP 413. Both limits apply before model inference; callers must reduce the encoded file size, downsample, or tile oversized inputs. The server serializes inference and rejects overlapping requests with HTTP 503. Its detector length is configured server-wide by OCR_INFER_LENGTH (640 or 1024, default 1024). NIM request limits and engine settings belong to the deployed version, not this local server configuration. Neither integration adds experimental text-dense cropping or sharpening; preprocessing remains caller-owned.

Local weights and the torchvision backbone are cached at /models/huggingface and /models/torch inside the container. The example mounts the repository’s models/ directory there; change the host mount to use another persistent cache. The compiled CUDA extension stays in the Docker image. Stop only the container you launched with docker stop xr-ai-ocr-manual. The example’s --rm removes the stopped container, not the mounted cache. Rebuild after source changes and rerun after configuration changes. Docker rejects an occupied name or port rather than replacing an existing service. Closing an SDK client stops neither the local service nor an externally deployed NIM.

To test transcription against a running endpoint, run from the repository root:

XR_AI_OCR_TEST_URL=http://localhost:8112 uv --config-file uv.toml run --project tests \
  pytest tests/test_ocr_server.py::test_live_ocr_equipment_text

This opt-in HTTP test does not require a local GPU or Docker and never owns containers. It skips when XR_AI_OCR_TEST_URL is unset; do not mark it gpu or run it through container-management fixtures. The smoke test checks transcription on a square image, not polygon placement. CPU tests cover the response coordinate contract with mocked predictions; they do not establish real-model normalization on non-square images.

Speaker-conditioned STT#

services/speaker-stt is a private local inference process. It shares Nemotron 3 Diarization and Multitalker Parakeet model weights across sessions, while keeping each session’s diarization history, enrollment and ASR caches separate. In conversation and hybrid modes, a session enrolls one speaker with the configured start phrase and releases that identity with the stop phrase. With diagnostics disabled, background voices do not reach the transcript as separate speakers after enrollment: the decoder receives the mixed audio with owner and interference masks and is conditioned on the owner. Optional diagnostics expose ignored transcripts for display only; they cannot query the agent or change enrollment. Before enrollment, each detected speaker has independent conditioned decoding and phrase-matching state. Someone already talking does not reserve ownership: another speaker can say the start phrase over that conversation and enroll when their own utterance completes. Non-speech background noise does not impose a silence gate. Each speaker’s existing inactivity interval ends their utterance; there is no requirement that every speaker become quiet together.

All complete start phrases finalized in one accepted audio step are considered before ownership changes. One match enrolls its speaker; multiple matches in that step enroll nobody and require a fresh attempt. A phrase finalized later cannot take over an existing owner. This is a streaming decision boundary, not a guarantee of distinguishing all acoustically simultaneous requests. Speaker labels are session-local identities; actual attribution and interference suppression require qualification on the intended microphones.

Interaction mode is independent of the selected ASR backend. While closed, wake_only and hybrid can authorize an attributed, wake-addressed one-off without enrolling its speaker. A unique exact start takes priority in hybrid mode; ambiguous same-step requests authorize nobody. A bare wake permits only that recognized speaker’s next utterance whose onset precedes the configured grace deadline. always_on accepts all recognized speakers without enrollment. Candidate transcripts themselves are not query authorization or diagnostics.

The process follows the model-service HTTP pattern and exposes health and model identity endpoints. A private WebSocket carries mono signed 16-bit PCM at 16 kHz and returns enrollment, authorized one-turn, and transcription events. It serializes inference because model conditioning is mutable. Opening a session establishes the capture timestamp of its first sample. Subsequent audio requests form one ordered, contiguous PCM stream; the service derives all later timestamps from the number of samples it has accepted. The caller must close and reopen the session if its capture or transport layer drops audio, because that layer owns the information needed to detect the loss. Utterances forced to end at the configured duration limit cannot authorize a closed-state turn, enroll, or release a speaker; an enrolled owner’s forced segment is still transcribed. Start and stop phrases can span utterances within the conversation phrase window, retaining separate enrollment fragments for each speaker. A truncated candidate cannot enroll from the tail of the same continuing speech episode; that speaker must end its episode before making another attempt.

The private protocol makes that ownership explicit: the opening JSON message supplies audio_origin_us once, and subsequent binary WebSocket messages supply only PCM bytes. Each connection owns one session, and the caller waits for the event response before sending the next audio message. Audio must be serialized in capture order. The service neither reorders audio nor guesses whether samples are missing from wall-clock arrival times. Binary messages retain the existing limit of one second of audio (16,000 complete signed 16-bit samples, or 32,000 bytes). Invalid message types, JSON, or incomplete PCM samples are protocol errors. The server’s standard transport message-size and heartbeat defaults are retained; JSON controls do not inherit an unrelated audio-duration limit.

The private identity is protocol 5, adding the explicit diagnostics setting to protocol 4’s interaction policy. Worker and speaker service must be restarted together after updating: earlier endpoints fail admission rather than silently selecting batch STT or accepting incompatible interaction settings.

Configure model IDs, device, precision, host, port and session capacity in services/speaker-stt/speaker_stt.yaml. The default HTTP base URL is http://127.0.0.1:8102. Keep the unauthenticated private streaming endpoint on a loopback or trusted interface. Hardware profiles or CUDA_VISIBLE_DEVICES select a GPU; the generic configuration uses the first visible GPU. From the repository root, start it with:

uv --config-file uv.toml run --project services/speaker-stt speaker_stt

The service warms both streaming and final decoding before signaling ready. Reuse verifies the local listener’s managed-process ownership and resolved configuration before launching a child; a changed configuration requires stopping that process first. A matching listener still warming its models refuses another launch rather than allocating a duplicate GPU copy. Persistent reuse exits after signaling readiness; monitored reuse keeps its readiness proxy alive.

Each session owns one diarization cache and at most eight attributed ASR continuations, allocated when their speaker identities appear. Features and model weights are shared. The Multitalker Parakeet encoder uses 160 ms chunks. Native encoder and predictor state continue through quiet intervals; utterance boundaries clear only the text being delivered. Accumulated native transcript metadata is replaced after each step while retaining its decoder state, so completed turns do not grow the hypothesis history. Initial speaker admission replays at most the encoder’s existing left-context window through that speaker’s masks. Always-on and wake-only keep candidate decoding as their steady state; hybrid uses it whenever no owner conversation is open. Every candidate remains required, so a decoder failure resets that session rather than silently ignoring a competing speaker. After enrollment only the owner’s ASR continuation is retained with diagnostics disabled. Enabled diagnostics share encoded features but add decoding work and separate continuations for non-owner speakers. Every retained continuation advances through actual quiet audio; this adds inference work compared with resetting each utterance. With the default max_sessions: 16, up to 128 enrollment decoder caches can coexist; GPU memory and latency at that bound have not been qualified. Candidate-decoding failure revokes the session because dropping a candidate could conceal a phrase tie. All cache allocation, inference, and release use one serialization boundary.

At a quiet endpoint, final text is decoded on an isolated copy with 320 ms of terminal silence. This model-specific padding allows delayed subword emissions to complete; it does not change the configured silence threshold or advance the live capture timeline. Forced utterance truncation does not add terminal silence. Tokens already delivered by finalization are reconciled against subsequent real decoding before new-turn text is exposed. If the real continuation revises that finalized suffix, the inference session fails and releases enrollment rather than assigning old words to a new turn. The padding is a bounded policy qualified with recorded and generated speech, not a universal recognition guarantee.

A connected session may pause without losing enrollment. Disconnect, transport liveness failure, or inference failure releases it; max_sessions bounds retained sessions. Cancelled model work finishes before its cache and serialization ownership are released, including repeated cancellation. The model environment is isolated from workers and the existing batch STT service. Model weights are downloaded on first use and then reused from the configured cache. As with the sibling NeMo services, existing HF_HOME and NEMO_CACHE_DIR values take precedence over cache defaults.

The WebSocket session is not a public model API. It is reserved for the private typed voice integration; applications continue to use the public voice and model interfaces rather than connecting to this route directly.

Two HuggingFace cache roots#

The servers use two different HF_HOME values, so HuggingFace weights live in two separate trees under the service’s resolved model_cache:

Consumer

HF_HOME

Hub cache

vLLM-backed servers (pip and docker)

<model_cache>/

<model_cache>/hub/

STT and Magpie TTS (NeMo host processes)

<model_cache>/huggingface/

<model_cache>/huggingface/hub/

(The NeMo servers additionally cache non-HF artifacts under <model_cache>/nemo/.) For host processes (pip-mode vLLM, STT, Magpie TTS) these are defaults: an HF_HOME or NEMO_CACHE_DIR already set in the environment takes priority. Docker-mode vLLM always uses the YAML model_cache, which it mounts into the container. Pocket TTS defaults HF_HOME to <model_cache>/pocket/huggingface/ and HF_XET_CACHE to <model_cache>/pocket/xet/.

model_cache itself is set per YAML and resolved relative to the YAML file. Both the model-servers profiles and the standalone service YAMLs resolve it to models/ at the repository root. For a manual hf download, set HF_HOME to match the consumer:

# vLLM-served model, launched via a model-servers profile
# (model_cache resolves to models/ at the repository root):
HF_HOME=models hf download nvidia/Cosmos3-Nano

# NeMo STT server launched from its standalone YAML:
HF_HOME=models/huggingface hf download nvidia/parakeet-tdt-0.6b-v3

Starting without network access#

The following model services look up their models in the Hugging Face cache before contacting the Hub, and load cached models from their local paths:

Service

Cached artifact

Loaded through

Docker-mode vLLM servers

Snapshot of model at its --revision (default main)

vllm serve <snapshot directory>, keeping the configured served model name

STT and Magpie TTS

<model-name>.nemo; Magpie uses model_revision when set

NeMo restore_from

Pocket TTS

Weights, tokenizer, and voice state named by the selected language

A cache-local copy of that language configuration

Clef

Snapshot at model_revision

Native release loader with the snapshot directory

The listed services use cached artifacts without refreshing them from the Hub. Only a cache miss follows the online download path. Pocket TTS accepts its ungated weights when the gated voice-cloning weights are not cached. Pip-mode vLLM passes the model ID to vLLM unchanged, so it can still contact the Hub.

Offline startup supports a previously working cache. A local snapshot lookup does not establish completeness or integrity. Interrupted or damaged caches may require an online re-download; automatic partial-cache repair is not provided. The underlying loader may fail or apply defaults for missing files. To recover, re-download the affected model and revision with hf download <model-id> --revision <revision> --force-download, using the same HF_HOME or --cache-dir as the service, then start the service again while online.

A cache hit is never refreshed. A model without a pinned revision keeps the main commit it was downloaded at; update it with hf download, as shown above.

After a successful online start, the listed services can reuse their working model cache without network access. This assumes their runtime dependencies and Docker images are already installed. To forbid downloads, set HF_HUB_OFFLINE=1. The host services inherit it, and the vLLM wrapper forwards it into its containers:

HF_HUB_OFFLINE=1 uv run --project agent-samples/simple-vlm-example/model-servers simple_vlm_example_model_servers

In this mode, a service whose model is not cached fails at startup with an error that names the model, its revision, and the cache directory. Like the other forwarded Hub settings, changing HF_HUB_OFFLINE recreates persistent vLLM containers.

Migrating model caches from ai-services/#

The service-directory rename does not move ignored model weights. Existing installations may still hold VLM, STT, and LLM weights in ai-services/models/, and TTS weights in ai-services/tts/models/; all current service profiles resolve to the repository-root models/ directory.

Stop model services, then merge the caches from the repository root:

mkdir -p models
if [ -d ai-services/models ]; then
  cp -a -l -n ai-services/models/. models/
fi
if [ -d ai-services/tts/models ]; then
  cp -a -l -n ai-services/tts/models/. models/
fi

The -l option avoids duplicating large weight files and requires both paths to use the same filesystem. Omit -l when copying across filesystems and verify free space first. Keep the old directories until the relocated services start successfully without network access. Recreate project environments with uv sync; do not copy .venv directories across the move.

Connecting a sample to shared model services#

Each sample’s model-servers/ project conveniently starts its required local inference services. Shared defaults live under utils/xr-ai-model-server-stack/; sample GPU profiles contain only placement and workload overrides. Workers consume endpoints from their models JSON and can also use services started individually or independently operated endpoints. Refer to Customizing model servers for configuration and the stop/start flow.

Declare only application processes in the sample orchestrator. Application capability services such as RAG remain sample-owned. For startup ordering and readiness, refer to Consumer model readiness.

Calling these from a worker#

Workers do not hand-roll httpx clients against these endpoints. They depend on xr-ai-models, load a per-sample model profile, and construct service clients via make_llm, make_vlm, make_stt, make_tts, make_embedding, and make_decision. The SDK encapsulates the OpenAI-compatible wire format and the per-model quirks (reasoning-field aliasing, chat_template_kwargs, served-model-name strings) so callers never branch on backend.

from xr_ai_models import load_models_config, make_llm, ChatMessage

config = load_models_config("yaml/models.json")
async with make_llm(config, "llm") as llm:
    resp = await llm.chat(
        [ChatMessage(role="user", content="hello")],
        max_tokens=128,
        enable_thinking=True,
    )
    print(resp.content, resp.reasoning)

A consumer model profile specifies adapter behavior and endpoint connectivity:

{
  "models": {
    "llm": {
      "category": "llm",
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    }
  }
}

The worker-side xr-ai-models loader accepts JSON or YAML, including flat legacy entries and direct role mappings. A profile shared with the launcher must use the wrapped nested .json contract; non-.json profiles are rejected before deployment metadata is read. Full protocol surface, the preset table, and the profile contract are in xr-ai-models.

Hosting models on NVIDIA NIM#

Develop with the local shared model servers and deploy the same workers against independently operated NIM endpoints. The model profile selects a protocol adapter and an endpoint; samples continue to use the typed xr_ai_models interfaces. There is no XR AI NIM container launcher or local HTTP proxy.

Chat and vision use openai_compat with native model IDs. Speech uses riva_grpc, which normalizes Riva audio inside the SDK. Asymmetric embedding endpoints use the existing adapter.default_extras.input_type field; the SDK translates query and passage labels without changing the RAG service.

Self-hosted NIM endpoints#

Operators own container lifecycle, GPU placement, optimized-engine caching, health exposure, and endpoint authorization. Consumer profiles use external ownership. The local model-servers launcher remains a development convenience. Refer to Deploying samples with NIM endpoints for a complete profile, sample configuration paths, smoke checks, and deployment qualification.

Model-server persistence#

The persistent vLLM-backed servers (vlm_server, nemotron3_nano_llm_server, nemotron_omni_llm_server, embedding_server) survive agent restarts by design: the sample’s convenience launcher starts them and returns after readiness, leaving weights warm. Use the sample’s model command with --stop for the selected local deployment. Startup does not sweep services omitted from its configuration. Independently deployed NIMs follow their operator’s lifecycle. At startup a wrapper that finds a different persistent xr-ai container holding its port (found by the xr-ai-vllm.port=<port> label) stops and removes it before launching its own. Each persistent wrapper script checks its health endpoint before spawning vLLM:

  • Already running with a matching launch fingerprint → touch the ready file immediately, then idle. Stack is ready in seconds; no model reload.

  • Matching container still starting → attach to its lifecycle and keep waiting for /health instead of issuing a conflicting second docker run.

  • Already running with changed or legacy configuration → stop, remove, and recreate the repository-owned container from the current YAML.

  • Healthy endpoint without the expected running container → fail without stopping the unowned listener.

  • Stopped Docker container with matching launch fingerprint → restart it, wait for /health, then touch the ready file.

  • Stopped Docker container with changed or legacy configuration → remove and recreate it from the current YAML before waiting for /health.

  • Not running → spawn vLLM normally, wait for /health, then touch the ready file.

In pip mode, vLLM is spawned with start_new_session=True so the launcher’s killpg() does not reach it on shutdown. In docker mode, Docker owns the container while the foreground docker run client uses its own session. Either way vLLM keeps running after the orchestrator exits.

The STT server follows the same pattern without Docker: stt_server spawns its persistent process with start_new_session=True, reuses a healthy server that survived a previous stack run, and is stopped by the sample’s model command with --stop when selected in that stack.

Most shipped model-server profiles that include voice select speaker ASR; procedure-capture-guide selects batch STT for narration without speaker enrollment. When selected, the speaker STT launcher waits for model loading, decoder warmup, and HTTP readiness before reporting ready, then reuses a warm instance with matching configuration. Configuration changes require stopping and restarting the service. --stop stops only the sample’s explicitly selected server list, using its configured ports and the existing process-ownership guards. The speaker service listens on port 8102 and supports streaming audio over a private WebSocket. Native NIM speech uses independently deployed Riva endpoints rather than this local model stack. Refer to Deploying samples with NIM endpoints and xr-ai-voice for worker backend selection and enrollment.

Pocket TTS uses the launcher’s persistent process group directly. Its bootstrap reuses an existing healthy listener. In a monitored stack, the reuse invocation remains alive as a health proxy so the launcher can detect service loss. A persistent-only launcher that uses exit_after_ready=True explicitly permits that proxy to exit after readiness, so repeated model-servers starts do not leave idle wrappers behind. When no server exists, the bootstrap replaces itself with the foreground Uvicorn and Pocket TTS process.

Docker containers carry a fingerprint of their image, GPU assignment, model cache, environment, bootstrap packages, complete vLLM command, and a versioned launcher-controlled Docker contract. This prevents a failed container created by one sample profile—or by older launcher behavior—from being restarted later with stale memory limits, entrypoint, setup commands, or model arguments.

Stopping the persisted servers, from the repo root:

uv run --project agent-samples/simple-vlm-example/model-servers simple_vlm_example_model_servers --stop

Cleanup locates labelled Docker containers before inspecting ports, then stops them with docker stop (escalating to docker kill after 20 s). Locally persisted processes (pip-mode vLLM and Pocket TTS) must carry the XR_AI_VLLM_MANAGED and XR_AI_VLLM_PORT ownership markers before cleanup sends SIGTERM or SIGKILL. Pocket TTS records the dedicated process group created by the launcher without leaving that group. Cleanup signals the complete group only after verifying both the listener marker and, when separate, the launcher’s group-leader marker, so inference descendants cannot survive as orphans. Launcher abort-time escalation can reach Pocket TTS before it signals readiness; once its ready file exists when cleanup begins, the launcher preserves it. Explicit stop falls back to PID-only cleanup if group ownership cannot be verified; other local servers always retain PID-only cleanup. Unknown listeners and failed inspection abort cleanup without sending a signal, and the sample’s model command with --stop exits nonzero if any target could not be stopped. Absent servers are silently skipped.

Stop targets are only the servers listed by that sample’s convenience launcher, with ports read from the service defaults and selected sample GPU overrides. Use the same explicit --gpu-profile NAME on start and stop. Stop does not require model-download credentials.

Before switching samples, updating the checkout, or editing server settings, stop the current stack using its current checkout and profile. Then start the new stack. The convenience launchers do not track previous configurations or provide isolation between samples using the same ports. Cleanup can stop a recognized service used by another sample or manual invocation. Refer to Customizing model servers for the complete development flow.

Choosing the vLLM runtime (pip vs Docker)#

All four vLLM-backed servers (vlm_server, nemotron3_nano_llm_server, nemotron_omni_llm_server, embedding_server) accept a vllm_backend: key in their YAML to pick how vLLM is hosted:

vllm_backend

Runtime

Code fallback

Shipped standalone YAMLs

Use when

pip

vllm serve from the wrapper’s venv

yes

no

Developing the wrapper in its local environment or using a custom pip installation.

docker

docker run <vllm_image> vllm serve …

no

yes

Running the configured vLLM container used by the checked-in configurations.

Both modes honor identical configuration keys — same model, same port, same vLLM flags. Docker-only lifecycle settings such as vllm_image, extra_pip, and spark_uma are accepted and ignored in pip mode. The dispatcher lives in utils/xr-ai-vllm/. Switching is one YAML edit:

# vlm-server (Cosmos3)
vllm_backend: docker
vllm_image:   nvcr.io/nvidia/vllm:26.10.pre1-py3

vllm_image: defaults to nvcr.io/nvidia/vllm:26.10.pre1-py3 for all wrappers. The Python environments also pin vLLM 0.31.0. vLLM 0.31.0 is the upstream fixed release for CVE-2026-114050 and CVE-2026-114055. Keep per-request multimodal processor overrides disabled; server-level processor configuration remains available.

This NVIDIA NGC preview uses CUDA 13.4 and reports vLLM 0.31.0+88641915 on Linux x86_64 and aarch64. The checked-in Cosmos3 and Nemotron model configurations select this image. Qualify the preview on the target GPU and model workload before deploying it. Nemotron Omni uses its native Mamba and causal-convolution implementations, so the shipped configuration does not compile or install mamba-ssm or causal-conv1d. Override the image to pin another tag, an internal mirror, or a custom build.

The preview tag is mutable. The observed multi-architecture manifest digest is sha256:cb38bbefdbc3e9fe045f931dd72cbd58cd48985c7e09916bcc6c15c10f39b929. For reproducible deployments, append @sha256:<digest> to the image reference in each service configuration and qualify that exact manifest.

Important

When upgrading an existing checkout, stop the persistent model stack before starting it with the new image:

uv run --project agent-samples/simple-vlm-example/model-servers simple_vlm_example_model_servers --stop
docker pull nvcr.io/nvidia/vllm:26.10.pre1-py3

The next launch recreates stale managed containers when their image or command fingerprint differs. After the new stack starts successfully, reclaim disk from an old image with docker image rm <old-vllm-image>. Keep the shared model cache; the updated stack reuses compatible weights and downloads any missing artifacts.

docker mode — prerequisites#

  • Docker Engine accessible from the launch session (docker ps must succeed without sudo). Refer to Docker access and NVIDIA runtime for group setup.

  • NVIDIA Container Toolkit installed, with its nvidia runtime registered in Docker. Refer to Docker access and NVIDIA runtime for the runtime check, configuration steps, and GPU smoke test.

  • NGC pull access, when the configured vllm_image is restricted or requires authentication on nvcr.io. The wrapper auto-runs docker login nvcr.io if NGC_API_KEY is in the environment (loaded by load_credentials() from ~/.config/xr-ai/credentials.json per Credentials). Otherwise, log in manually once:

    docker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY

Existing ~/.docker/config.json entries take priority and are not overwritten.

docker mode — runtime details#

  • Container is launched with --network host --ipc host --runtime nvidia (forwarding NVIDIA_VISIBLE_DEVICES), and /bin/bash overrides the image entrypoint so setup installs run before vllm serve.

  • Failed stopped containers are recreated because Docker cannot change their recorded entrypoint or command.

  • The host model_cache is bind-mounted at the same path inside the container and HF_HOME is set to it, so weights cached by pip mode are reused by docker mode and vice versa.

  • Before vllm serve starts, the container resolves the model snapshot from the mounted cache, as described in Starting without network access. Only when the snapshot must be downloaded does the wrapper verify that the image’s Hub exposes its Xet download path and has hf-xet>=1.1.2,<2.0.0. It repairs a missing or incompatible hf-xet wheel and checks the complete integration again; an unavailable accelerator therefore fails startup instead of silently falling back to plain HTTPS. The download completes and is synced to disk before vLLM initializes CUDA.

  • HF_XET_HIGH_PERFORMANCE=1 is the default in pip and docker modes. Setting it to 0 disables high-performance tuning, not Xet itself. Set HF_HUB_DISABLE_XET=1 to disable Xet. To use legacy hf_transfer with a compatible Hub version, install the hf_transfer package and set both HF_HUB_DISABLE_XET=1 and HF_HUB_ENABLE_HF_TRANSFER=1; Xet-backed files continue to use Xet when it remains enabled. These values are preserved, forwarded, and included in the container fingerprint, so changing one recreates a persistent container.

The selected NGC preview reports this vLLM version in its image metadata:

Image

vLLM

nvcr.io/nvidia/vllm:26.10.pre1-py3

0.31.0+88641915

  • Container name is deterministic per service: xr-ai-vllm-vlm-server, xr-ai-vllm-nemotron3-nano-llm-server, xr-ai-vllm-nemotron-omni-llm-server, and xr-ai-vllm-embedding-server.

  • Persistence parity: all four vLLM-backed wrappers launch their Docker processes in separate sessions, so they survive launcher shutdowns like their pip-mode start_new_session=True counterparts.

Cleanup#

the sample’s model command with --stop works for both modes within the selected managed launch plan. Cleanup locates labelled Docker containers before inspecting ports, then stops them with docker stop (escalating to docker kill after 20 s). Pip-mode processes carry an xr-ai-vllm ownership marker; unknown listeners and failed inspection abort cleanup without sending a signal.

Pip-mode vLLM processes started before the ownership markers were introduced cannot be identified safely. After upgrading, stop each unmarked process manually once; subsequent launches include the markers and support managed cleanup.

Per-server notes#

  • vlm-server defaults to the Cosmos3 Nano Reasoner. Hugging Face publishes Reasoner and Generator weights in the unified nvidia/Cosmos3-Nano checkpoint. The required Cosmos3ForConditionalGeneration architecture override selects vLLM’s native Reasoner loader: despite its generic class name, it maps only the understanding tower and vision encoder and drops the Generator weights. The Generator requires vLLM’s separate --omni path, which xr-ai intentionally does not enable. The checkpoint’s official chat template emits the assistant answer directly and has no enable_thinking branch or <think> delimiters, so the client preset needs no reasoning-field mapping. Cosmos-Reason1 remains available by pairing model: nvidia/Cosmos-Reason1-7B with the cosmos_vlm client preset. Hosting backend is selectable per YAML — refer to Choosing the vLLM runtime above. When serving nvidia/Cosmos3-Nano with the default Docker image on SM12 GPUs, including RTX Blackwell and GB10, the service selects Triton attention for both the decoder and vision encoder. Measurements on NGC 26.09 showed lower cold image latency with Triton: the x86 image fell back to FlashAttention-2 PTX compilation, while the ARM image included native Blackwell kernels. This conservative default is retained for the NGC 26.10 preview pending GPU qualification; those kernel properties and latency results have not been reverified on the preview. Other GPUs, custom images, and pip mode retain vLLM’s backend selection. The optional attention_backend and mm_encoder_attn_backend YAML keys override each component independently; explicit auto restores vLLM’s selection. Cold starts still include model loading and backend compilation. A healthy model HTTP endpoint is not a guarantee that sample inference warmup has completed.

  • nemotron3-nano-llm is a thin wrapper around vllm serve for NVIDIA-Nemotron-3-Nano-30B-A3B-{NVFP4,FP8} (auto-selected by GPU compute capability). vLLM handles tool calling (qwen3_coder parser), reasoning extraction (nano_v3 parser — auto-fetched into model_cache), and FlashInfer FP4 MoE kernels. model_blackwell selects the NVFP4 checkpoint on SM100+; model_ada selects FP8 on earlier supported GPUs. Parsed reasoning is returned in the reasoning field. The nemotron3_nano client preset disables thinking by default so short calls retain an answer token budget; pass enable_thinking=True on a call to opt in. Native FP4 requires a Blackwell-class GPU such as B200 or RTX PRO 6000; FP8 is used on Ada, Hopper, and Ampere. enforce_eager: true by default to avoid the silent 3–8 min CUDA graph and FlashInfer autotune on cold start. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime). The model card recommends temperature=1.0 and top_p=1.0 for reasoning, and temperature=0.6 and top_p=0.95 for tool calling. It identifies the model as ready for commercial use under the NVIDIA Nemotron Open Model License.

  • nemotron-omni-llm is a vLLM-backed multimodal LLM serving Nemotron-3-Nano-Omni-30B-A3B-Reasoning (text + video input) at port 8108. The YAML auto-selects between three model variants by detected GPU compute capability: NVFP4 on Blackwell (SM100+), FP8 on Ada, Hopper, and Ampere, BF16 forced via use_bf16: true for highest quality at the largest VRAM cost. Same OpenAI-compatible HTTP contract as the other LLM servers — swap the port to swap backends. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime); persists across stack restarts in both pip and docker modes.

  • stt-server loads parakeet-tdt-0.6b-v3 via NeMo ASR in-process. The language and temperature form fields are accepted but ignored; recognition uses the loaded model’s language behavior. Set startup_timeout_s to a positive finite number to override the 600-second cold-start budget.

  • magpie-tts loads magpie_tts_multilingual_357m via NeMo TTS in-process.

  • pocket-tts loads the compact kyutai/pocket-tts model on the configured cpu, cuda, or automatically selected device. The checked-in deployment profiles use CUDA and warm up the model before reporting ready. Its native PCM streaming path emits audio during generation; non-streaming WAV requests remain supported. Model loading and synthesis run outside the asyncio event loop. The default bill_boerst voice derives from a CC0 Voice-Zero recording and is the only voice accepted by this release. The service logs whether it loaded the gated voice-cloning weights or the ungated fallback.

    To stream, send POST /v1/audio/speech with response_format set to pcm and stream set to true. The response uses audio/pcm; the x-audio-sample-rate and x-audio-channels headers describe the signed 16-bit interleaved samples. Streaming with another response format returns HTTP 400.

  • embedding-server serves nvidia/Nemotron-3-Embed-1B-BF16 through /v1/embeddings under the served name embed. It emits 2048-dimensional vectors. Retrieval inputs use query: for queries and passage: for documents, as specified in the model card. The standalone configuration reserves 20% of a GPU and uses Docker; the tea sample applies its own memory budget and indexes all 2048 dimensions. Its Spark profile selects Triton attention for GB10 compatibility; other profiles leave attention-backend selection to vLLM. Refer to Release migration for the model upgrade steps.

  • rag-service is a typed dense document-retrieval capability. Point documents_dir at an application-owned tree and models_config at a profile with an embedding role. It chunks and embeds supported documents at startup, caches its index, and returns matches above min_score. Start the embedding service first, RAG second, and the consuming worker last. The cache includes document content, indexing settings, the resolved embedding-role hash, the server-reported root from the matching /v1/models entry, and optional cache_key. Unrelated model-role edits do not invalidate the cache. The reported identity is best-effort: this repository’s Docker launcher serves a resolved snapshot path, while pip serves the Hugging Face repository ID. Switching backends can rebuild the cache, and changing a Hub revision under the same pip model ID requires a new cache_key. A served alias such as embed does not identify weights. Providers omitting root cause documents to be re-embedded on every startup without persistent caching; cache_key cannot opt back in. Failed metadata requests or invalid responses abort startup. The manual key supplements the role hash and reported identity. Every replica must remain on the same weights throughout indexing and serving; health probes check availability, not model consistency. Restart RAG whenever the embedding model changes. Refer to Release migration for upgrading existing indexes and servers.

  • video-memory-service owns recorded chunk queries, NVDEC, and PNG output behind typed msgpack over ZMQ on port 8310. Set recordings_dir in its YAML to enable recorded-video operations; the path must match the hub’s video_recording.out_dir. Latest video and sampling windows end at the newest recorded timestamp and require only a duration. Historical frame, video, and sampling requests share an absolute start_us; video windows add a duration. Sampling also accepts a hard total frame budget, decodes each selected chunk once, skips unavailable or corrupt chunks when other frames remain, and can bound exported PNG dimensions. The sampled timestamps are estimates interpolated from chunk metadata. The selection budget may be up to 256, but the shipped Cosmos VLM accepts no more than four selected images per inference request. Current frames stay with the caller’s hub client. When recordings_dir is empty, participant discovery returns an empty list and recorded-media operations return recording_disabled.

  • Ports are configurable — avoid conflicts with LiveKit (7880–7882) and hub (8080, 8090).

  • Standalone service YAMLs live beside services for direct single-service launches; they are not sample configuration. Sample convenience stacks start from those service YAMLs and apply overrides under agent-samples/<sample>/model-servers/yaml/<gpu-profile>.yaml. Workers consume the resulting endpoints through their models JSON.