AI inference servers#

Read this when calling or operating an inference server. For the orchestrator pattern that wires servers into a sample, refer to Adding a new sample. For an end-to-end procedure covering custom deployment profiles, hardware YAML, and consumer model configuration, refer to Customizing model servers.

Multiple reusable inference and typed capability services are available as launchable peers of services/device-io-hub/. Model HTTP is encapsulated by the typed factories in xr-ai-models; workers must not add vendor SDKs or hand-written HTTP clients. Reference services cover vision-language reasoning, speech recognition, text-to-speech, embeddings, large language models, recorded video, and document retrieval.

Server

Command

Port

Model

Backend

services/vlm-server/

vlm_server

8100

Cosmos3 Nano Reasoner

vLLM (pip or docker)

services/stt-server/

stt_server

8103

parakeet-tdt-0.6b-v3

NeMo ASR in-process

services/magpie-tts/

magpie_tts_server

8104

magpie_tts_multilingual_357m

NeMo TTS in-process

services/pocket-tts/

pocket_tts_server

8105

kyutai/pocket-tts

Pocket TTS in-process

services/llama-nemotron-llm/

llama_nemotron_llm_server

8106

Llama-3.1-Nemotron-Nano-8B-v1

vLLM (pip or docker)

services/nemotron3-nano-llm/

nemotron3_nano_llm_server

8107

NVIDIA-Nemotron-3-Nano-30B-A3B-{NVFP4,FP8}

vLLM (pip or docker)

services/nemotron-omni-llm/

nemotron_omni_llm_server

8108

Nemotron-3-Nano-Omni-30B-A3B-Reasoning (NVFP4, FP8, or BF16, GPU-selected)

vLLM (pip or docker) — multimodal (text + video)

services/embedding-server/

embedding_server

8109

llama-nemotron-embed-1b-v2

vLLM (pip or docker)

services/nim-server/

nim_server

configured per YAML

selected NVIDIA NIM

persistent Docker container

services/video-memory-service/

video_memory_service

8310

—

Typed recorded-video capability

services/rag-service/

rag_service

8340

—

Typed dense document retrieval capability

Local model artifacts land in the service’s model_cache directory, set per YAML and resolved relative to the YAML file. Self-hosted NIM containers use nim_cache for their optimized engines and weights; hosted endpoints keep no weights in this repository. Every models/ tree is excluded from version control. The model-servers profiles share models/ at the repository root; the exact layout per launch style is below.

Two HuggingFace cache roots#

The servers use two different HF_HOME values, so HuggingFace weights live in two separate trees under the service’s resolved model_cache:

Consumer

HF_HOME

Hub cache

vLLM-backed servers (pip and docker)

<model_cache>/

<model_cache>/hub/

STT and Magpie TTS (NeMo host processes)

<model_cache>/huggingface/

<model_cache>/huggingface/hub/

(The NeMo servers additionally cache non-HF artifacts under <model_cache>/nemo/.) For host processes (pip-mode vLLM, STT, Magpie TTS) these are defaults: an HF_HOME or NEMO_CACHE_DIR already set in the environment takes priority. Docker-mode vLLM always uses the YAML model_cache, which it mounts into the container. Pocket TTS defaults HF_HOME to <model_cache>/pocket/huggingface/ and HF_XET_CACHE to <model_cache>/pocket/xet/.

model_cache itself is set per YAML and resolved relative to the YAML file. Both the model-servers profiles and the standalone service YAMLs resolve it to models/ at the repository root. For a manual hf download, set HF_HOME to match the consumer:

# vLLM-served model, launched via a model-servers profile
# (model_cache resolves to models/ at the repository root):
HF_HOME=models hf download nvidia/Cosmos3-Nano

# NeMo STT server launched from its standalone YAML:
HF_HOME=models/huggingface hf download nvidia/parakeet-tdt-0.6b-v3

Migrating model caches from ai-services/#

The service-directory rename does not move ignored model weights. Existing installations may still hold VLM, STT, and LLM weights in ai-services/models/, and TTS weights in ai-services/tts/models/; all current service profiles resolve to the repository-root models/ directory.

Stop model services, then merge the caches from the repository root:

mkdir -p models
if [ -d ai-services/models ]; then
  cp -a -l -n ai-services/models/. models/
fi
if [ -d ai-services/tts/models ]; then
  cp -a -l -n ai-services/tts/models/. models/
fi

The -l option avoids duplicating large weight files and requires both paths to use the same filesystem. Omit -l when copying across filesystems and verify free space first. Keep the old directories until the relocated services start successfully without network access. Recreate project environments with uv sync; do not copy .venv directories across the move.

Connecting a sample to shared model services#

Model services are operator-owned by the persistent model-servers stack. Application samples reuse those endpoints; they do not own model processes or copy server YAML into their own configuration directories. Follow Customizing model servers to select a deployment profile, tune its hardware-specific server YAML, and copy compatible endpoint entries into the sample’s models JSON. Omit the deployment object from copied entries; it belongs to the shared model-server profile.

Declare only application processes in the sample orchestrator. Application capability services such as RAG remain sample-owned. For startup ordering and readiness, refer to Consumer model readiness.

Calling these from a worker#

Workers do not hand-roll httpx clients against these endpoints. They depend on xr-ai-models, load a per-sample model profile, and construct service clients via make_llm, make_vlm, make_stt, make_tts, and make_embedding. The SDK encapsulates the OpenAI-compatible wire format and the per-model quirks (reasoning-field aliasing, chat_template_kwargs, served-model-name strings) so callers never branch on backend.

from xr_ai_models import load_models_config, make_llm, ChatMessage

config = load_models_config("yaml/models.json")
async with make_llm(config, "agent_llm") as llm:
    resp = await llm.chat(
        [ChatMessage(role="user", content="hello")],
        max_tokens=128,
        enable_thinking=True,
    )
    print(resp.content, resp.reasoning)

A consumer model profile specifies adapter behavior and endpoint connectivity:

{
  "models": {
    "agent_llm": {
      "category": "llm",
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    }
  }
}

The worker-side xr-ai-models loader accepts JSON or YAML, including flat legacy entries and direct role mappings. A profile shared with the stdlib-only launcher must use the wrapped nested .json contract; non-.json profiles are rejected before deployment metadata is read. Full protocol surface, the preset table, and the profile contract are in xr-ai-models.

Hosting models on NVIDIA NIM#

The LLM and VLM can run on NVIDIA NIM instead of local vLLM — NIM exposes the same OpenAI-compatible /v1/chat/completions API, so this is a model-profile change with no worker code edits. STT and TTS stay local: hosted NIM speech (Riva) is not OpenAI /v1/audio-compatible. Self-hosted speech NIMs are covered below.

A hosted consumer entry uses an environment-variable reference for its credential and omits deployment and health polling metadata:

{
  "models": {
    "vlm": {
      "category": "vlm",
      "adapter": {
        "kind": "openai_compat",
        "model_name": "nvidia/cosmos3-nano-reasoner",
        "capabilities": {"vision": true, "streaming": true}
      },
      "endpoint": {
        "base_url": "https://integrate.api.nvidia.com",
        "api_key_env": "NGC_API_KEY"
      }
    }
  }
}

  • api_key_env: NGC_API_KEY sends the environment value as a bearer token. The key is a managed credential — run_stack injects a saved NGC_API_KEY into every subprocess (refer to Credentials); or export it.

  • Omitted deployment defaults to an externally operated endpoint; no model process is started or stopped by the profile.

  • model_name is the hosted model id from build.nvidia.com.

To adapt a sample, copy its active model profile, replace the local model entry with the hosted entry, and point models_config in the worker YAML at the new file. The worker reads the endpoint credential named in api_key_env; export it or configure the credential store. Consumer sample launchers do not inspect model profiles to choose model processes.

Self-hosted NIM containers (models.vlm_llm_nim.json)#

Compatible models can be pulled from NGC and served as optimized NIM containers on your own GPUs: same APIs as hosted NIM and no network hop.

The NIM containers are owned by the shared model-servers stack, exactly like the local vLLM servers. Its deployment profiles pick the mix: every managed entry launches as a nim_server process (services/nim-server, a generic wrapper; the per-GPU-profile nim_<role>_server.yaml picks the image and ports) or as a local server:

uv run --project model-server-samples/model-servers model_servers --models vlm_llm_nim

  • vlm_llm_nim: Nemotron-3 Nano Omni and Cosmos3-Nano Reasoner as NIM containers, with STT, Pocket TTS, and embedding served locally. Samples reuse these endpoints; they never launch or stop the containers.

To configure a sample with these endpoints, refer to Customizing model servers.

The container image: is the model, so swapping models is a nim_<role>_server.yaml edit plus the matching profile entry. Selection is per entry, not per profile: each model role independently picks a local server, a self-hosted NIM container, or a hosted endpoint through its own adapter and endpoint sections. The model-server launcher also requires deployment metadata; consumer profiles omit it. The shipped profiles are presets, not a closed set; a mixed setup is a copy of a shipped profile with the relevant entries changed, saved under any name and selected with --models (model-servers) or models_config (workers). When mixing, mind port overlaps: give a NIM container a free port or drop the overlapping local server from the profile.

A custom model-server profile can still launch self-hosted Riva speech NIMs. Workers reach them through the optional riva_grpc model kind:

stt:
  kind:      riva_grpc
  category:  stt
  base_url:  localhost:50051   # the container's gRPC port
  language:  en-US

TTS additionally takes voice: (a Riva voice name) and sample_rate: (default 44100). An explicit health() call with health_check: true (the default) runs a gRPC channel-ready probe; consumer startup does not call it. No shipped model-server profile or sample selects Riva speech.

Requirements: docker + NVIDIA Container Toolkit, NGC_API_KEY (used for the nvcr.io image pull and by the container itself to download the GPU-matched optimized engine from NGC on first start; multi-GB, cached under models/nim/ for later runs), and GPU capacity for every container. cuda_visible_devices placement lives in the per-GPU-profile nim_*_server.yaml files. Their adjacent comments record profile-specific validation status and hardware cautions; verify startup and capacity on the target host. Readiness gates on each container’s /v1/health/ready.

A NIM container serving something the samples don’t ship is the same mechanism by hand: point an openai_compat entry’s base_url at its port or a riva_grpc entry at its gRPC port. Consumer workers do not require a health route. Operators can still check the container’s /v1/health/ready endpoint directly. With ownership: external (you run the container yourself) that is the whole change. For an orchestrator to launch or expect it, the entry’s deployment.service must name a process row in that orchestrator’s service table (_MODEL_SERVICES in model-servers, _MODEL_PROCESSES in a sample); a service name with no row fails fast at startup, and adding one row plus its config YAML is the only orchestrator edit the profile system ever needs.

Model-server persistence#

The persistent vLLM-backed servers (vlm_server, llama_nemotron_llm_server, nemotron3_nano_llm_server, nemotron_omni_llm_server, embedding_server) and self-hosted NIM containers (nim_server) survive stack restarts by design, including when a deployment profile marks them managed: the stack starts them, but a clean shutdown leaves them serving so the next start reuses hot weights. model_servers --stop is the teardown. Switching profiles needs no manual teardown: at startup a wrapper that finds a different persistent xr-ai container holding its port (found by the xr-ai-vllm.port=<port> label) stops and removes it before launching its own. Each persistent wrapper script checks its health endpoint before spawning vLLM:

  • Already running with a matching launch fingerprint → touch the ready file immediately, then idle. Stack is ready in seconds; no model reload.

  • Matching container still starting → attach to its lifecycle and keep waiting for /health instead of issuing a conflicting second docker run.

  • Already running with changed or legacy configuration → stop, remove, and recreate the repository-owned container from the current YAML.

  • Healthy endpoint without the expected running container → fail without stopping the unowned listener.

  • Stopped Docker container with matching launch fingerprint → restart it, wait for /health, then touch the ready file.

  • Stopped Docker container with changed or legacy configuration → remove and recreate it from the current YAML before waiting for /health.

  • Not running → spawn vLLM normally, wait for /health, then touch the ready file.

In pip mode, vLLM is spawned with start_new_session=True so the launcher’s killpg() does not reach it on shutdown. In docker mode, Docker owns the container while the foreground docker run client uses its own session. Either way vLLM keeps running after the orchestrator exits.

The STT server follows the same pattern without Docker: stt_server spawns its persistent process with start_new_session=True, reuses a healthy server that survived a previous stack run, and is stopped by the same model_servers --stop cleanup.

Pocket TTS uses the launcher’s persistent process group directly. Its bootstrap reuses an existing healthy listener. In a monitored stack, the reuse invocation remains alive as a health proxy so the launcher can detect service loss. A persistent-only launcher that uses exit_after_ready=True explicitly permits that proxy to exit after readiness, so repeated model-servers starts do not leave idle wrappers behind. When no server exists, the bootstrap replaces itself with the foreground Uvicorn and Pocket TTS process.

Docker containers carry a fingerprint of their image, GPU assignment, model cache, environment, bootstrap packages, complete vLLM command, and a versioned launcher-controlled Docker contract. This prevents a failed container created by one sample profile—or by older launcher behavior—from being restarted later with stale memory limits, entrypoint, setup commands, or model arguments.

Stopping the persisted servers, from the repo root:

uv run --project model-server-samples/model-servers model_servers --stop

Cleanup locates labelled Docker containers before inspecting ports, then stops them with docker stop (escalating to docker kill after 20 s). Locally persisted processes (pip-mode vLLM and Pocket TTS) must carry the XR_AI_VLLM_MANAGED and XR_AI_VLLM_PORT ownership markers before cleanup sends SIGTERM or SIGKILL. Pocket TTS records the dedicated process group created by the launcher without leaving that group. Cleanup signals the complete group only after verifying both the listener marker and, when separate, the launcher’s group-leader marker, so inference descendants cannot survive as orphans and launcher abort-time escalation can still reach Pocket TTS. It falls back to PID-only cleanup if group ownership cannot be verified; other local servers always retain PID-only cleanup. Unknown listeners and failed inspection abort cleanup without sending a signal, and model_servers --stop exits nonzero if any target could not be stopped. Absent servers are silently skipped.

The target ports and container names match the defaults in the per-profile YAML files.

Choosing the vLLM runtime (pip vs Docker)#

All five vLLM-backed servers (vlm_server, llama_nemotron_llm_server, nemotron3_nano_llm_server, nemotron_omni_llm_server, embedding_server) accept a vllm_backend: key in their YAML to pick how vLLM is hosted:

vllm_backend

Runtime

Code fallback

Shipped standalone YAMLs

Use when

pip

vllm serve from the wrapper’s venv

yes

no

Developing the wrapper in its local environment or using a custom pip installation.

docker

docker run <vllm_image> vllm serve …

no

yes

Running the configured vLLM container used by the checked-in configurations.

Both modes honor identical configuration keys — same model, same port, same vLLM flags. Docker-only lifecycle settings such as vllm_image, extra_pip, and spark_uma are accepted and ignored in pip mode. The dispatcher lives in utils/xr-ai-vllm/. Switching is one YAML edit:

# vlm-server (Cosmos3)
vllm_backend: docker
vllm_image:   nvcr.io/nvidia/vllm:26.08-py3

vllm_image: defaults to nvcr.io/nvidia/vllm:26.08-py3 for all wrappers. This image includes vLLM 0.27.1 and supports the checked-in Cosmos3 and Nemotron model configurations. Nemotron Omni uses the image’s native Mamba and causal-convolution implementations, so the shipped configuration does not compile or install mamba-ssm or causal-conv1d. Override the image to pin another tag, an internal mirror, or a custom build.

Important

When upgrading an existing checkout, stop the persistent model stack before starting it with the new image:

uv run --project model-server-samples/model-servers model_servers --stop
docker pull nvcr.io/nvidia/vllm:26.08-py3

The next launch recreates stale managed containers when their image or command fingerprint differs. After the new stack starts successfully, reclaim disk from an old image with docker image rm <old-vllm-image>. Keep the shared model cache; the 26.08 stack reuses compatible weights and downloads any missing artifacts.

docker mode — prerequisites#

  • Docker Engine accessible from the launch session (docker ps must succeed without sudo). Refer to Docker access and NVIDIA runtime for group setup.

  • NVIDIA Container Toolkit installed, with its nvidia runtime registered in Docker. Refer to Docker access and NVIDIA runtime for the runtime check, configuration steps, and GPU smoke test.

  • NGC pull access, when the configured vllm_image is restricted or requires authentication on nvcr.io. The wrapper auto-runs docker login nvcr.io if NGC_API_KEY is in the environment (loaded by load_credentials() from ~/.config/xr-ai/credentials.json per Credentials). Otherwise, log in manually once:

    docker login nvcr.io -u '$oauthtoken' -p $NGC_API_KEY

Existing ~/.docker/config.json entries take priority and are not overwritten.

docker mode — runtime details#

  • Container is launched with --network host --ipc host --runtime nvidia (forwarding NVIDIA_VISIBLE_DEVICES), and /bin/bash overrides the image entrypoint so setup installs run before vllm serve.

  • Failed stopped containers are recreated because Docker cannot change their recorded entrypoint or command.

  • The host model_cache is bind-mounted at the same path inside the container and HF_HOME is set to it, so weights cached by pip mode are reused by docker mode and vice versa.

  • Before vllm serve starts, the wrapper verifies that the image’s Hub exposes its Xet download path and has hf-xet>=1.1.2,<2.0.0. It repairs a missing or incompatible hf-xet wheel and checks the complete integration again; an unavailable accelerator therefore fails startup instead of silently falling back to plain HTTPS.

  • HF_XET_HIGH_PERFORMANCE=1 is the default in pip and docker modes. Setting it to 0 disables high-performance tuning, not Xet itself. Set HF_HUB_DISABLE_XET=1 to disable Xet. To use legacy hf_transfer with a compatible Hub version, install the hf_transfer package and set both HF_HUB_DISABLE_XET=1 and HF_HUB_ENABLE_HF_TRANSFER=1; Xet-backed files continue to use Xet when it remains enabled. These values are preserved, forwarded, and included in the container fingerprint, so changing one recreates a persistent container.

The shipped image pin was qualified with this in-container vLLM version:

Image

vLLM

nvcr.io/nvidia/vllm:26.08-py3

0.27.1

  • Container name is deterministic per service: xr-ai-vllm-vlm-server, xr-ai-vllm-llama-nemotron-llm-server, xr-ai-vllm-nemotron3-nano-llm-server, xr-ai-vllm-nemotron-omni-llm-server, and xr-ai-vllm-embedding-server.

  • Persistence parity: all five vLLM-backed wrappers launch their Docker processes in separate sessions, so they survive launcher shutdowns like their pip-mode start_new_session=True counterparts.

Cleanup#

model_servers --stop works for both modes. Cleanup locates labelled Docker containers before inspecting ports, then stops them with docker stop (escalating to docker kill after 20 s). Pip-mode processes carry an xr-ai-vllm ownership marker; unknown listeners and failed inspection abort cleanup without sending a signal.

Pip-mode vLLM processes started before the ownership markers were introduced cannot be identified safely. After upgrading, stop each unmarked process manually once; subsequent launches include the markers and support managed cleanup.

Per-server notes#

  • vlm-server defaults to the Cosmos3 Nano Reasoner. Hugging Face publishes Reasoner and Generator weights in the unified nvidia/Cosmos3-Nano checkpoint. The required Cosmos3ForConditionalGeneration architecture override selects vLLM’s native Reasoner loader: despite its generic class name, it maps only the understanding tower and vision encoder and drops the Generator weights. The Generator requires vLLM’s separate --omni path, which xr-ai intentionally does not enable. The checkpoint’s official chat template emits the assistant answer directly and has no enable_thinking branch or <think> delimiters, so the client preset needs no reasoning-field mapping. Cosmos-Reason1 remains available by pairing model: nvidia/Cosmos-Reason1-7B with the cosmos_vlm client preset. Hosting backend is selectable per YAML — refer to Choosing the vLLM runtime above.

  • llama-nemotron-llm is a thin wrapper around vllm serve for Llama-3.1-Nemotron-Nano-8B-v1. vLLM handles native Llama-3.1 tool calling via the llama3_json parser — tools=[...] in the request is rendered via the model’s chat template and the resulting tool calls come back in OpenAI wire format (finish_reason: "tool_calls"). Per-turn reasoning toggle via "detailed thinking on" or "detailed thinking off" in a system or user message. Without either phrase, this model reasons by default; the reasoning preamble is not stripped server-side. To swap checkpoints, change model only to one with a compatible chat template, and update tool_call_parser for the replacement model’s tool syntax. Revisit max_model_len, gpu_memory_utilization, and tensor_parallel_size for its context and GPU footprint. Keep served_model_name: llm to retain the built-in llama_nemotron adapter, or update the model profile’s adapter when changing that name or any wire behavior. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime). The model card recommends temperature=0.6 and top_p=0.95 with reasoning enabled, and greedy decoding with reasoning disabled. It identifies the model as ready for commercial use under the NVIDIA Open Model License and the Llama 3.1 Community License.

  • nemotron3-nano-llm is a thin wrapper around vllm serve for NVIDIA-Nemotron-3-Nano-30B-A3B-{NVFP4,FP8} (auto-selected by GPU compute capability). vLLM handles tool calling (qwen3_coder parser), reasoning extraction (nano_v3 parser — auto-fetched into model_cache), and FlashInfer FP4 MoE kernels. model_blackwell selects the NVFP4 checkpoint on SM100+; model_ada selects FP8 on earlier supported GPUs. Parsed reasoning is returned in the reasoning field. The nemotron3_nano client preset disables thinking by default so short calls retain an answer token budget; pass enable_thinking=True on a call to opt in. Native FP4 requires a Blackwell-class GPU such as B200 or RTX PRO 6000; FP8 is used on Ada, Hopper, and Ampere. enforce_eager: true by default to avoid the silent 3–8 min CUDA graph and FlashInfer autotune on cold start. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime). The model card recommends temperature=1.0 and top_p=1.0 for reasoning, and temperature=0.6 and top_p=0.95 for tool calling. It identifies the model as ready for commercial use under the NVIDIA Nemotron Open Model License.

  • nemotron-omni-llm is a vLLM-backed multimodal LLM serving Nemotron-3-Nano-Omni-30B-A3B-Reasoning (text + video input) at port 8108. The YAML auto-selects between three model variants by detected GPU compute capability: NVFP4 on Blackwell (SM100+), FP8 on Ada, Hopper, and Ampere, BF16 forced via use_bf16: true for highest quality at the largest VRAM cost. Same OpenAI-compatible HTTP contract as the other LLM servers — swap the port to swap backends. Hosting backend is selectable per YAML (refer to Choosing the vLLM runtime); persists across stack restarts in both pip and docker modes.

  • stt-server loads parakeet-tdt-0.6b-v3 via NeMo ASR in-process. English-only; the language and temperature form fields are accepted but ignored. Set startup_timeout_s to a positive finite number to override the 600-second cold-start budget.

  • magpie-tts loads magpie_tts_multilingual_357m via NeMo TTS in-process.

  • pocket-tts loads the compact kyutai/pocket-tts model on the configured cpu, cuda, or automatically selected device. The checked-in deployment profiles use CUDA and warm up the model before reporting ready. Its native PCM streaming path emits audio during generation; non-streaming WAV requests remain supported. Model loading and synthesis run outside the asyncio event loop. The default bill_boerst voice derives from a CC0 Voice-Zero recording and is the only voice accepted by this release. The service logs whether it loaded the gated voice-cloning weights or the ungated fallback.

    To stream, send POST /v1/audio/speech with response_format set to pcm and stream set to true. The response uses audio/pcm; the x-audio-sample-rate and x-audio-channels headers describe the signed 16-bit interleaved samples. Streaming with another response format returns HTTP 400.

  • embedding-server serves nvidia/llama-nemotron-embed-1b-v2 through /v1/embeddings. It emits 2048-dimensional Matryoshka embeddings and can truncate them to 384, 512, 768, 1024, or 2048 dimensions. The checked-in configuration reserves 20% of a GPU and uses Docker.

  • rag-service is a typed dense document-retrieval capability. Point documents_dir at an application-owned tree and models_config at a profile with an embedding role. It chunks and embeds supported documents at startup, caches its index, and returns matches above min_score. Start the embedding service first, RAG second, and the consuming worker last. The cache includes document content, indexing settings, and the model profile; change cache_key when a remote endpoint changes its backing model without changing that profile.

  • video-memory-service owns recorded chunk queries, NVDEC, and PNG output behind typed msgpack over ZMQ on port 8310. Set recordings_dir in its YAML to enable recorded-video operations; the path must match the hub’s video_recording.out_dir. Latest video and sampling windows end at the newest recorded timestamp and require only a duration. Historical frame, video, and sampling requests share an absolute start_us; video windows add a duration. Sampling also accepts a hard total frame budget, decodes each selected chunk once, skips unavailable or corrupt chunks when other frames remain, and can bound exported PNG dimensions. The sampled timestamps are estimates interpolated from chunk metadata. The selection budget may be up to 256, but the shipped Cosmos VLM accepts no more than four selected images per inference request. Current frames stay with the caller’s hub client. When recordings_dir is empty, participant discovery returns an empty list and recorded-media operations return recording_disabled.

  • Ports are configurable — avoid conflicts with LiveKit (7880–7882) and hub (8080, 8090).

  • Standalone service YAMLs live beside services for direct single-service launches; they are not sample configuration. Shared deployments use the hardware-specific YAML under model-server-samples/model-servers/yaml/<gpu-profile>/, while samples reuse the resulting endpoints through their models JSON.

  • The generic NIM wrapper has no service-local YAML. Use a hardware profile under model-server-samples/model-servers/yaml/<gpu-profile>/; its nim_<role>_server.yaml files use nim_cache, normally ../../../../models/nim from that location.