AI inference servers#

Read this when calling or operating an inference server. For the orchestrator pattern that wires servers into a sample, refer to Adding a new sample. For an end-to-end procedure covering custom deployment profiles, hardware YAML, and consumer model configuration, refer to Customizing model servers.

Multiple reusable inference and typed capability services are available as launchable peers of services/device-io-hub/. Model HTTP is encapsulated by the typed factories in xr-ai-models; workers must not add vendor SDKs or hand-written HTTP clients. Reference services cover vision-language reasoning, speech recognition, text-to-speech, embeddings, large language models, recorded video, and document retrieval.

Server

Command

Port

Model

Backend

services/vlm-server/

vlm_server

8100

Cosmos3 Nano Reasoner

vLLM (pip or docker)

services/stt-server/

stt_server

8103

parakeet-tdt-0.6b-v3

NeMo ASR in-process

services/speaker-stt/

speaker_stt

8102

Nemotron 3 Diarization + Multitalker Parakeet

NeMo streaming ASR

services/magpie-tts/

magpie_tts_server

8104

magpie_tts_multilingual_357m

NeMo TTS in-process

services/magpie-nim-tts/

magpie_nim_tts

8105

Magpie speech NIM

HTTP adapter over Riva gRPC

services/pocket-tts/

pocket_tts_server

8105

kyutai/pocket-tts

Pocket TTS in-process

services/llama-nemotron-llm/

llama_nemotron_llm_server

8106

Llama-3.1-Nemotron-Nano-8B-v1

vLLM (pip or docker)

services/nemotron3-nano-llm/

nemotron3_nano_llm_server

8107

NVIDIA-Nemotron-3-Nano-30B-A3B-{NVFP4,FP8}

vLLM (pip or docker)

services/nemotron-omni-llm/

nemotron_omni_llm_server

8108

Nemotron-3-Nano-Omni-30B-A3B-Reasoning (NVFP4, FP8, or BF16, GPU-selected)

vLLM (pip or docker) — multimodal (text + video)

services/clef-server/

clef_server via model-server-samples/clef-flash/

8120

Cloudflare Clef-Flash 9B

Native SystemOne choice decisions

services/embedding-server/

embedding_server

8109

llama-nemotron-embed-1b-v2

vLLM (pip or docker)

services/nim-server/

nim_server

configured per YAML

selected NVIDIA NIM

persistent Docker container

services/video-memory-service/

video_memory_service

8310

—

Typed recorded-video capability

services/rag-service/

rag_service

8340

—

Typed dense document retrieval capability

Local model artifacts land in the service’s model_cache directory, set per YAML and resolved relative to the YAML file. Self-hosted NIM containers use nim_cache for their optimized engines and weights; hosted endpoints keep no weights in this repository. Every models/ tree is excluded from version control. The model-servers profiles share models/ at the repository root; the exact layout per launch style is below.

Speaker-conditioned STT#

services/speaker-stt is a private local inference process. It shares Nemotron 3 Diarization and Multitalker Parakeet model weights across sessions, while keeping each session’s diarization history, enrollment and ASR caches separate. A session enrolls one speaker with the configured start phrase and releases that identity with the stop phrase. Background voices do not reach the transcript as separate speakers after enrollment: the decoder receives the mixed audio with owner and interference masks and is conditioned on the owner. Before enrollment, each detected speaker has independent conditioned decoding and phrase-matching state. Someone already talking does not reserve ownership: another speaker can say the start phrase over that conversation and enroll when their own utterance completes. Non-speech background noise does not impose a silence gate. Each speaker’s existing inactivity interval ends their utterance; there is no requirement that every speaker become quiet together.

All complete start phrases finalized in one accepted audio step are considered before ownership changes. One match enrolls its speaker; multiple matches in that step enroll nobody and require a fresh attempt. A phrase finalized later cannot take over an existing owner. This is a streaming decision boundary, not a guarantee of distinguishing all acoustically simultaneous requests. Speaker labels are session-local identities; actual attribution and interference suppression require qualification on the intended microphones.

The process follows the model-service HTTP pattern and exposes health and model identity endpoints. A private WebSocket carries mono signed 16-bit PCM at 16 kHz and returns enrollment and transcription events. It serializes inference because model conditioning is mutable. Opening a session establishes the capture timestamp of its first sample. Subsequent audio requests form one ordered, contiguous PCM stream; the service derives all later timestamps from the number of samples it has accepted. The caller must close and reopen the session if its capture or transport layer drops audio, because that layer owns the information needed to detect the loss. Utterances forced to end at the configured duration limit cannot enroll or release a speaker; an enrolled owner’s forced segment is still transcribed. Start and stop phrases can span utterances within the conversation phrase window, retaining separate enrollment fragments for each speaker. A truncated candidate cannot enroll from the tail of the same continuing speech episode; that speaker must end its episode before making another attempt.

The private protocol makes that ownership explicit: the opening JSON message supplies audio_origin_us once, and subsequent binary WebSocket messages supply only PCM bytes. Each connection owns one session, and the caller waits for the event response before sending the next audio message. Audio must be serialized in capture order. The service neither reorders audio nor guesses whether samples are missing from wall-clock arrival times. Binary messages retain the existing limit of one second of audio (16,000 complete signed 16-bit samples, or 32,000 bytes). Invalid message types, JSON, or incomplete PCM samples are protocol errors. The server’s standard transport message-size and heartbeat defaults are retained; JSON controls do not inherit an unrelated audio-duration limit.

Configure model IDs, device, precision, host, port and session capacity in services/speaker-stt/speaker_stt.yaml. The default HTTP base URL is http://127.0.0.1:8102. Keep the unauthenticated private streaming endpoint on a loopback or trusted interface. Hardware profiles or CUDA_VISIBLE_DEVICES select a GPU; the generic configuration uses the first visible GPU. From the repository root, start it with:

uv --config-file uv.toml run --project services/speaker-stt speaker_stt

The service warms both streaming and final decoding before signaling ready. Reuse verifies the local listener’s managed-process ownership and resolved configuration before launching a child; a changed configuration requires stopping that process first. A matching listener still warming its models refuses another launch rather than allocating a duplicate GPU copy. Persistent reuse exits after signaling readiness; monitored reuse keeps its readiness proxy alive.

Each session owns one diarization cache and at most eight temporary ASR caches before enrollment, bounded by the model’s speaker-label capacity. Features and model weights are shared. After enrollment only the owner’s ASR cache is needed unless the dependent diagnostics layer requests other transcripts. With the default max_sessions: 16, up to 128 enrollment decoder caches can coexist; GPU memory and latency at that bound have not been qualified. Candidate-decoding failure revokes the session because dropping a candidate could conceal a phrase tie. All cache allocation, inference, and release use one serialization boundary. A connected session may pause without losing enrollment. Disconnect, transport liveness failure, or inference failure releases it; max_sessions bounds retained sessions. Cancelled model work finishes before its cache and serialization ownership are released, including repeated cancellation. The model environment is isolated from workers and the existing batch STT service. Model weights are downloaded on first use and then reused from the configured cache. As with the sibling NeMo services, existing HF_HOME and NEMO_CACHE_DIR values take precedence over cache defaults.

The WebSocket session is not a public model API. It is reserved for the private typed voice integration; applications continue to use the public voice and model interfaces rather than connecting to this route directly.

Two HuggingFace cache roots#

The servers use two different HF_HOME values, so HuggingFace weights live in two separate trees under the service’s resolved model_cache:

Consumer

HF_HOME

Hub cache

vLLM-backed servers (pip and docker)

<model_cache>/

<model_cache>/hub/

STT and Magpie TTS (NeMo host processes)

<model_cache>/huggingface/

<model_cache>/huggingface/hub/

(The NeMo servers additionally cache non-HF artifacts under <model_cache>/nemo/.) For host processes (pip-mode vLLM, STT, Magpie TTS) these are defaults: an HF_HOME or NEMO_CACHE_DIR already set in the environment takes priority. Docker-mode vLLM always uses the YAML model_cache, which it mounts into the container. Pocket TTS defaults HF_HOME to <model_cache>/pocket/huggingface/ and HF_XET_CACHE to <model_cache>/pocket/xet/.

model_cache itself is set per YAML and resolved relative to the YAML file. Both the model-servers profiles and the standalone service YAMLs resolve it to models/ at the repository root. For a manual hf download, set HF_HOME to match the consumer:

# vLLM-served model, launched via a model-servers profile
# (model_cache resolves to models/ at the repository root):
HF_HOME=models hf download nvidia/Cosmos3-Nano

# NeMo STT server launched from its standalone YAML:
HF_HOME=models/huggingface hf download nvidia/parakeet-tdt-0.6b-v3

Starting without network access#

The following model services look up their models in the Hugging Face cache before contacting the Hub, and load cached models from their local paths:

Service

Cached artifact

Loaded through

Docker-mode vLLM servers

Snapshot of model at its --revision (default main)

vllm serve <snapshot directory>, keeping the configured served model name

STT and Magpie TTS

<model-name>.nemo; Magpie uses model_revision when set

NeMo restore_from

Pocket TTS

Weights, tokenizer, and voice state named by the selected language

A cache-local copy of that language configuration

Clef

Snapshot at model_revision

Native release loader with the snapshot directory

The listed services use cached artifacts without refreshing them from the Hub. Only a cache miss follows the online download path. Pocket TTS accepts its ungated weights when the gated voice-cloning weights are not cached. Pip-mode vLLM passes the model ID to vLLM unchanged, so it can still contact the Hub.

Offline startup supports a previously working cache. A local snapshot lookup does not establish completeness or integrity. Interrupted or damaged caches may require an online re-download; automatic partial-cache repair is not provided. The underlying loader may fail or apply defaults for missing files. To recover, re-download the affected model and revision with hf download <model-id> --revision <revision> --force-download, using the same HF_HOME or --cache-dir as the service, then start the service again while online.

A cache hit is never refreshed. A model without a pinned revision keeps the main commit it was downloaded at; update it with hf download, as shown above.

After a successful online start, the listed services can reuse their working model cache without network access. This assumes their runtime dependencies and Docker images are already installed. To forbid downloads, set HF_HUB_OFFLINE=1. The host services inherit it, and the vLLM wrapper forwards it into its containers:

HF_HUB_OFFLINE=1 uv run --project model-server-samples/model-servers model_servers

In this mode, a service whose model is not cached fails at startup with an error that names the model, its revision, and the cache directory. Like the other forwarded Hub settings, changing HF_HUB_OFFLINE recreates persistent vLLM containers.

Migrating model caches from ai-services/#

The service-directory rename does not move ignored model weights. Existing installations may still hold VLM, STT, and LLM weights in ai-services/models/, and TTS weights in ai-services/tts/models/; all current service profiles resolve to the repository-root models/ directory.

Stop model services, then merge the caches from the repository root:

mkdir -p models
if [ -d ai-services/models ]; then
  cp -a -l -n ai-services/models/. models/
fi
if [ -d ai-services/tts/models ]; then
  cp -a -l -n ai-services/tts/models/. models/
fi

The -l option avoids duplicating large weight files and requires both paths to use the same filesystem. Omit -l when copying across filesystems and verify free space first. Keep the old directories until the relocated services start successfully without network access. Recreate project environments with uv sync; do not copy .venv directories across the move.

Connecting a sample to shared model services#

Model services are operator-owned by the persistent model-servers stack. Application samples reuse those endpoints; they do not own model processes or copy server YAML into their own configuration directories. Follow Customizing model servers to select a deployment profile, tune its hardware-specific server YAML, and copy compatible endpoint entries into the sample’s models JSON. Omit the deployment object from copied entries; it belongs to the shared model-server profile.

Declare only application processes in the sample orchestrator. Application capability services such as RAG remain sample-owned. For startup ordering and readiness, refer to Consumer model readiness.

Calling these from a worker#

Workers do not hand-roll httpx clients against these endpoints. They depend on xr-ai-models, load a per-sample model profile, and construct service clients via make_llm, make_vlm, make_stt, make_tts, make_embedding, and make_decision. The SDK encapsulates the OpenAI-compatible wire format and the per-model quirks (reasoning-field aliasing, chat_template_kwargs, served-model-name strings) so callers never branch on backend.

from xr_ai_models import load_models_config, make_llm, ChatMessage

config = load_models_config("yaml/models.json")
async with make_llm(config, "agent_llm") as llm:
    resp = await llm.chat(
        [ChatMessage(role="user", content="hello")],
        max_tokens=128,
        enable_thinking=True,
    )
    print(resp.content, resp.reasoning)

A consumer model profile specifies adapter behavior and endpoint connectivity:

{
  "models": {
    "agent_llm": {
      "category": "llm",
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    }
  }
}

The worker-side xr-ai-models loader accepts JSON or YAML, including flat legacy entries and direct role mappings. A profile shared with the stdlib-only launcher must use the wrapped nested .json contract; non-.json profiles are rejected before deployment metadata is read. Full protocol surface, the preset table, and the profile contract are in xr-ai-models.

Hosting models on NVIDIA NIM#

The LLM and VLM can run on NVIDIA NIM instead of local vLLM — NIM exposes the same OpenAI-compatible /v1/chat/completions API, so this is a model-profile change with no worker code edits. STT and TTS stay local: hosted NIM speech (Riva) is not OpenAI /v1/audio-compatible. Self-hosted speech NIMs are covered below.

A hosted consumer entry uses an environment-variable reference for its credential and omits deployment and health polling metadata:

{
  "models": {
    "vlm": {
      "category": "vlm",
      "adapter": {
        "kind": "openai_compat",
        "model_name": "nvidia/cosmos3-nano-reasoner",
        "capabilities": {"vision": true, "streaming": true}
      },
      "endpoint": {
        "base_url": "https://integrate.api.nvidia.com",
        "api_key_env": "NGC_API_KEY"
      }
    }
  }
}

  • api_key_env: NGC_API_KEY sends the environment value as a bearer token. The key is a managed credential — run_stack injects a saved NGC_API_KEY into every subprocess (refer to Credentials); or export it.

  • Omitted deployment defaults to an externally operated endpoint; no model process is started or stopped by the profile.

  • model_name is the hosted model id from build.nvidia.com.

To adapt a sample, copy its active model profile, replace the local model entry with the hosted entry, and point models_config in the worker YAML at the new file. The worker reads the endpoint credential named in api_key_env; export it or configure the credential store. Consumer sample launchers do not inspect model profiles to choose model processes.

Self-hosted NIM containers#

Compatible models can be pulled from NGC and served as optimized NIM containers on your own GPUs. Use the separate Shared model servers with NVIDIA NIM sample:

uv run --project model-server-samples/model-servers-nim model_servers_nim

Its hardware profiles select the container images, ports, and GPU placement; its compatibility adapters preserve the endpoints consumed by existing agent samples. The model-servers sample launches local server wrappers only. Refer to the NIM sample guide for model selection, hardware qualification, cache behavior, and shutdown commands.

Self-hosted Riva speech NIMs also expose a direct gRPC interface. Workers reach them through the optional riva_grpc model kind:

stt:
  kind:      riva_grpc
  category:  stt
  base_url:  localhost:50051   # the container's gRPC port
  language:  en-US

TTS additionally takes voice: (a Riva voice name) and sample_rate: (default 44100). An explicit health() call with health_check: true (the default) runs a gRPC channel-ready probe; consumer startup does not call it. The standalone Magpie NIM HTTP speech service service exposes a Magpie Riva NIM through the Pocket-compatible HTTP endpoint. Direct Riva clients require the optional SDK dependency; HTTP consumers do not.

Requirements: docker + NVIDIA Container Toolkit, NGC_API_KEY (used for the nvcr.io image pull and by the container itself to download the GPU-matched optimized engine from NGC on first start; multi-GB, cached under models/nim/ for later runs), and GPU capacity for every container. cuda_visible_devices placement lives in the per-GPU-profile nim_*_server.yaml files. Their adjacent comments record profile-specific validation status and hardware cautions; verify startup and capacity on the target host. Readiness gates on each container’s /v1/health/ready.

A NIM container serving something the samples don’t ship is the same mechanism by hand: point an openai_compat entry’s base_url at its port or a riva_grpc entry at its gRPC port. Consumer workers do not require a health route. Operators can still check the container’s /v1/health/ready endpoint directly. With ownership: external (you run the container yourself) that is the whole change. For an orchestrator to launch or expect it, the entry’s deployment.service must name a process row in that orchestrator’s service table; a service name with no row fails fast at startup, and adding one row plus its config YAML is the only orchestrator edit the profile system ever needs.

Model-server persistence#

The persistent vLLM-backed servers (vlm_server, llama_nemotron_llm_server, nemotron3_nano_llm_server, nemotron_omni_llm_server, embedding_server) and self-hosted NIM containers (nim_server) survive stack restarts by design, including when a deployment profile marks them managed: the stack starts them, but a clean shutdown leaves them serving so the next start reuses hot weights. Use model_servers --stop for the local stack or model_servers_nim --stop for the NIM stack. When switching between these samples, stop the current stack first. At startup a wrapper that finds a different persistent xr-ai container holding its port (found by the xr-ai-vllm.port=<port> label) stops and removes it before launching its own. Each persistent wrapper script checks its health endpoint before spawning vLLM:

  • Already running with a matching launch fingerprint → touch the ready file immediately, then idle. Stack is ready in seconds; no model reload.

  • Matching container still starting → attach to its lifecycle and keep waiting for /health instead of issuing a conflicting second docker run.

  • Already running with changed or legacy configuration → stop, remove, and recreate the repository-owned container from the current YAML.

  • Healthy endpoint without the expected running container → fail without stopping the unowned listener.

  • Stopped Docker container with matching launch fingerprint → restart it, wait for /health, then touch the ready file.

  • Stopped Docker container with changed or legacy configuration → remove and recreate it from the current YAML before waiting for /health.

  • Not running → spawn vLLM normally, wait for /health, then touch the ready file.

In pip mode, vLLM is spawned with start_new_session=True so the launcher’s killpg() does not reach it on shutdown. In docker mode, Docker owns the container while the foreground docker run client uses its own session. Either way vLLM keeps running after the orchestrator exits.

The STT server follows the same pattern without Docker: stt_server spawns its persistent process with start_new_session=True, reuses a healthy server that survived a previous stack run, and is stopped by the same model_servers --stop cleanup.

Pocket TTS uses the launcher’s persistent process group directly. Its bootstrap reuses an existing healthy listener. In a monitored stack, the reuse invocation remains alive as a health proxy so the launcher can detect service loss. A persistent-only launcher that uses exit_after_ready=True explicitly permits that proxy to exit after readiness, so repeated model-servers starts do not leave idle wrappers behind. When no server exists, the bootstrap replaces itself with the foreground Uvicorn and Pocket TTS process.

Docker containers carry a fingerprint of their image, GPU assignment, model cache, environment, bootstrap packages, complete vLLM command, and a versioned launcher-controlled Docker contract. This prevents a failed container created by one sample profile—or by older launcher behavior—from being restarted later with stale memory limits, entrypoint, setup commands, or model arguments.

Stopping the persisted servers, from the repo root:

uv run --project model-server-samples/model-servers model_servers --stop

Cleanup locates labelled Docker containers before inspecting ports, then stops them with docker stop (escalating to docker kill after 20 s). Locally persisted processes (pip-mode vLLM and Pocket TTS) must carry the XR_AI_VLLM_MANAGED and XR_AI_VLLM_PORT ownership markers before cleanup sends SIGTERM or SIGKILL. Pocket TTS records the dedicated process group created by the launcher without leaving that group. Cleanup signals the complete group only after verifying both the listener marker and, when separate, the launcher’s group-leader marker, so inference descendants cannot survive as orphans and launcher abort-time escalation can still reach Pocket TTS. It falls back to PID-only cleanup if group ownership cannot be verified; other local servers always retain PID-only cleanup. Unknown listeners and failed inspection abort cleanup without sending a signal, and model_servers --stop exits nonzero if any target could not be stopped. Absent servers are silently skipped.

The target ports and container names match the defaults in the per-profile YAML files.

Choosing the vLLM runtime (pip vs Docker)#

All five vLLM-backed servers (vlm_server, llama_nemotron_llm_server, nemotron3_nano_llm_server, nemotron_omni_llm_server, embedding_server) accept a vllm_backend: key in their YAML to pick how vLLM is hosted:

vllm_backend

Runtime

Code fallback

Shipped standalone YAMLs

Use when

pip

vllm serve from the wrapper’s venv

yes

no

Developing the wrapper in its local environment or using a custom pip installation.

docker

docker run <vllm_image> vllm serve …

no

yes

Running the configured vLLM container used by the checked-in configurations.

Both modes honor identical configuration keys — same model, same port, same vLLM flags. Docker-only lifecycle settings such as vllm_image, extra_pip, and spark_uma are accepted and ignored in pip mode. The dispatcher lives in utils/xr-ai-vllm/. Switching is one YAML edit:

# vlm-server (Cosmos3)
vllm_backend: docker
vllm_image:   nvcr.io/nvidia/vllm:26.09-py3

vllm_image: defaults to nvcr.io/nvidia/vllm:26.09-py3 for all wrappers. This image includes NVIDIA-patched vLLM 0.29.0. The checked-in Cosmos3 and Nemotron model configurations select this image. Nemotron Omni uses its native Mamba and causal-convolution implementations, so the shipped configuration does not compile or install mamba-ssm or causal-conv1d. Override the image to pin another tag, an internal mirror, or a custom build.

Important

When upgrading an existing checkout, stop the persistent model stack before starting it with the new image:

uv run --project model-server-samples/model-servers model_servers --stop
docker pull nvcr.io/nvidia/vllm:26.09-py3

The next launch recreates stale managed containers when their image or command fingerprint differs. After the new stack starts successfully, reclaim disk from an old image with docker image rm <old-vllm-image>. Keep the shared model cache; the 26.09 stack reuses compatible weights and downloads any missing artifacts.

docker mode — prerequisites#

  • Docker Engine accessible from the launch session (docker ps must succeed without sudo). Refer to Docker access and NVIDIA runtime for group setup.

  • NVIDIA Container Toolkit installed, with its nvidia runtime registered in Docker. Refer to Docker access and NVIDIA runtime for the runtime check, configuration steps, and GPU smoke test.

  • NGC pull access, when the configured vllm_image is restricted or requires authentication on nvcr.io. The wrapper auto-runs docker login nvcr.io if NGC_API_KEY is in the environment (loaded by load_credentials() from ~/.config/xr-ai/credentials.json per Credentials). Otherwise, log in manually once:

    docker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY

Existing ~/.docker/config.json entries take priority and are not overwritten.

docker mode — runtime details#

  • Container is launched with --network host --ipc host --runtime nvidia (forwarding NVIDIA_VISIBLE_DEVICES), and /bin/bash overrides the image entrypoint so setup installs run before vllm serve.

  • Failed stopped containers are recreated because Docker cannot change their recorded entrypoint or command.

  • The host model_cache is bind-mounted at the same path inside the container and HF_HOME is set to it, so weights cached by pip mode are reused by docker mode and vice versa.

  • Before vllm serve starts, the container resolves the model snapshot from the mounted cache, as described in Starting without network access. Only when the snapshot must be downloaded does the wrapper verify that the image’s Hub exposes its Xet download path and has hf-xet>=1.1.2,<2.0.0. It repairs a missing or incompatible hf-xet wheel and checks the complete integration again; an unavailable accelerator therefore fails startup instead of silently falling back to plain HTTPS. The download completes and is synced to disk before vLLM initializes CUDA.

  • HF_XET_HIGH_PERFORMANCE=1 is the default in pip and docker modes. Setting it to 0 disables high-performance tuning, not Xet itself. Set HF_HUB_DISABLE_XET=1 to disable Xet. To use legacy hf_transfer with a compatible Hub version, install the hf_transfer package and set both HF_HUB_DISABLE_XET=1 and HF_HUB_ENABLE_HF_TRANSFER=1; Xet-backed files continue to use Xet when it remains enabled. These values are preserved, forwarded, and included in the container fingerprint, so changing one recreates a persistent container.

The selected image reports this in-container vLLM version:

Image

vLLM

nvcr.io/nvidia/vllm:26.09-py3

0.29.0+5013de39

  • Container name is deterministic per service: xr-ai-vllm-vlm-server, xr-ai-vllm-llama-nemotron-llm-server, xr-ai-vllm-nemotron3-nano-llm-server, xr-ai-vllm-nemotron-omni-llm-server, and xr-ai-vllm-embedding-server.

  • Persistence parity: all five vLLM-backed wrappers launch their Docker processes in separate sessions, so they survive launcher shutdowns like their pip-mode start_new_session=True counterparts.

Cleanup#

model_servers --stop works for both modes. Cleanup locates labelled Docker containers before inspecting ports, then stops them with docker stop (escalating to docker kill after 20 s). Pip-mode processes carry an xr-ai-vllm ownership marker; unknown listeners and failed inspection abort cleanup without sending a signal.

Pip-mode vLLM processes started before the ownership markers were introduced cannot be identified safely. After upgrading, stop each unmarked process manually once; subsequent launches include the markers and support managed cleanup.

Per-server notes#

  • vlm-server defaults to the Cosmos3 Nano Reasoner. Hugging Face publishes Reasoner and Generator weights in the unified nvidia/Cosmos3-Nano checkpoint. The required Cosmos3ForConditionalGeneration architecture override selects vLLM’s native Reasoner loader: despite its generic class name, it maps only the understanding tower and vision encoder and drops the Generator weights. The Generator requires vLLM’s separate --omni path, which xr-ai intentionally does not enable. The checkpoint’s official chat template emits the assistant answer directly and has no enable_thinking branch or <think> delimiters, so the client preset needs no reasoning-field mapping. Cosmos-Reason1 remains available by pairing model: nvidia/Cosmos-Reason1-7B with the cosmos_vlm client preset. Hosting backend is selectable per YAML — refer to Choosing the vLLM runtime above.

  • llama-nemotron-llm is a thin wrapper around vllm serve for Llama-3.1-Nemotron-Nano-8B-v1. vLLM handles native Llama-3.1 tool calling via the llama3_json parser — tools=[...] in the request is rendered via the model’s chat template and the resulting tool calls come back in OpenAI wire format (finish_reason: "tool_calls"). Per-turn reasoning toggle via "detailed thinking on" or "detailed thinking off" in a system or user message. Without either phrase, this model reasons by default; the reasoning preamble is not stripped server-side. To swap checkpoints, change model only to one with a compatible chat template, and update tool_call_parser for the replacement model’s tool syntax. Revisit max_model_len, gpu_memory_utilization, and tensor_parallel_size for its context and GPU footprint. Keep served_model_name: llm to retain the built-in llama_nemotron adapter, or update the model profile’s adapter when changing that name or any wire behavior. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime). The model card recommends temperature=0.6 and top_p=0.95 with reasoning enabled, and greedy decoding with reasoning disabled. It identifies the model as ready for commercial use under the NVIDIA Open Model License and the Llama 3.1 Community License.

  • nemotron3-nano-llm is a thin wrapper around vllm serve for NVIDIA-Nemotron-3-Nano-30B-A3B-{NVFP4,FP8} (auto-selected by GPU compute capability). vLLM handles tool calling (qwen3_coder parser), reasoning extraction (nano_v3 parser — auto-fetched into model_cache), and FlashInfer FP4 MoE kernels. model_blackwell selects the NVFP4 checkpoint on SM100+; model_ada selects FP8 on earlier supported GPUs. Parsed reasoning is returned in the reasoning field. The nemotron3_nano client preset disables thinking by default so short calls retain an answer token budget; pass enable_thinking=True on a call to opt in. Native FP4 requires a Blackwell-class GPU such as B200 or RTX PRO 6000; FP8 is used on Ada, Hopper, and Ampere. enforce_eager: true by default to avoid the silent 3–8 min CUDA graph and FlashInfer autotune on cold start. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime). The model card recommends temperature=1.0 and top_p=1.0 for reasoning, and temperature=0.6 and top_p=0.95 for tool calling. It identifies the model as ready for commercial use under the NVIDIA Nemotron Open Model License.

  • nemotron-omni-llm is a vLLM-backed multimodal LLM serving Nemotron-3-Nano-Omni-30B-A3B-Reasoning (text + video input) at port 8108. The YAML auto-selects between three model variants by detected GPU compute capability: NVFP4 on Blackwell (SM100+), FP8 on Ada, Hopper, and Ampere, BF16 forced via use_bf16: true for highest quality at the largest VRAM cost. Same OpenAI-compatible HTTP contract as the other LLM servers — swap the port to swap backends. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime); persists across stack restarts in both pip and docker modes.

  • stt-server loads parakeet-tdt-0.6b-v3 via NeMo ASR in-process. English-only; the language and temperature form fields are accepted but ignored. Set startup_timeout_s to a positive finite number to override the 600-second cold-start budget.

  • magpie-tts loads magpie_tts_multilingual_357m via NeMo TTS in-process.

  • pocket-tts loads the compact kyutai/pocket-tts model on the configured cpu, cuda, or automatically selected device. The checked-in deployment profiles use CUDA and warm up the model before reporting ready. Its native PCM streaming path emits audio during generation; non-streaming WAV requests remain supported. Model loading and synthesis run outside the asyncio event loop. The default bill_boerst voice derives from a CC0 Voice-Zero recording and is the only voice accepted by this release. The service logs whether it loaded the gated voice-cloning weights or the ungated fallback.

    To stream, send POST /v1/audio/speech with response_format set to pcm and stream set to true. The response uses audio/pcm; the x-audio-sample-rate and x-audio-channels headers describe the signed 16-bit interleaved samples. Streaming with another response format returns HTTP 400.

  • embedding-server serves nvidia/llama-nemotron-embed-1b-v2 through /v1/embeddings. It emits 2048-dimensional Matryoshka embeddings and can truncate them to 384, 512, 768, 1024, or 2048 dimensions. The checked-in configuration reserves 20% of a GPU and uses Docker.

  • rag-service is a typed dense document-retrieval capability. Point documents_dir at an application-owned tree and models_config at a profile with an embedding role. It chunks and embeds supported documents at startup, caches its index, and returns matches above min_score. Start the embedding service first, RAG second, and the consuming worker last. The cache includes document content, indexing settings, and the model profile; change cache_key when a remote endpoint changes its backing model without changing that profile.

  • video-memory-service owns recorded chunk queries, NVDEC, and PNG output behind typed msgpack over ZMQ on port 8310. Set recordings_dir in its YAML to enable recorded-video operations; the path must match the hub’s video_recording.out_dir. Latest video and sampling windows end at the newest recorded timestamp and require only a duration. Historical frame, video, and sampling requests share an absolute start_us; video windows add a duration. Sampling also accepts a hard total frame budget, decodes each selected chunk once, skips unavailable or corrupt chunks when other frames remain, and can bound exported PNG dimensions. The sampled timestamps are estimates interpolated from chunk metadata. The selection budget may be up to 256, but the shipped Cosmos VLM accepts no more than four selected images per inference request. Current frames stay with the caller’s hub client. When recordings_dir is empty, participant discovery returns an empty list and recorded-media operations return recording_disabled.

  • Ports are configurable — avoid conflicts with LiveKit (7880–7882) and hub (8080, 8090).

  • Standalone service YAMLs live beside services for direct single-service launches; they are not sample configuration. Shared deployments use the hardware-specific YAML under model-server-samples/model-servers/yaml/<gpu-profile>/, while samples reuse the resulting endpoints through their models JSON.

  • The generic NIM wrapper has no service-local YAML. Use a hardware profile under model-server-samples/model-servers-nim/yaml/<gpu-profile>/; its nim_<role>_server.yaml files use nim_cache, normally ../../../../models/nim from that location.