Sample configuration reference#

Overview#

Checked-in YAML and JSON configuration for samples and model services, including example values and field comments. Use the catalog to locate settings before editing them; the operational guides explain when and how to apply changes.

Configuration catalog#

This reference is generated directly from configuration files under each top-level sample’s yaml/ and model-servers/yaml/ directories in agent-samples/, and from configuration files beside a direct capability subproject. Adding a file in these locations enrolls it automatically. The catalog also includes service YAML files under services/<service>/ that declare model_cache; these supply the shared model-server defaults. Values are sample- and hardware-specific examples, not universal defaults. Keep field guidance beside the value as a source comment; the generated page preserves those comments verbatim.

Refer to the generated Python API reference for public typed model configuration fields. Operational choices, credentials, and deployment workflows remain in the handwritten guides.

To change a sample parameter:

Start in agent-samples/<sample>/ and find the owning file in that sample’s README or published guide.
Edit the checked-in YAML or JSON value, preserving its documented type. Resolve relative paths from the file that declares them unless the sample guide documents different precedence.
Restart the sample process that owns the file. Sample configuration is not hot-reloaded.
If the edit changes a persistent model server, stop and restart the shared model stack. Refer to Customizing model servers for that workflow.

conversation-example#

model-servers/yaml/96G_blackwell.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Other settings come from each service's YAML under services/.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 8
  gpu_memory_utilization: 0.35
  kv_cache_memory_bytes: 2147483648
vlm:
  gpu_memory_utilization: 0.23
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4
clef: {}

model-servers/yaml/dual_48G_ada.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Other settings come from each service's YAML under services/.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '1'
tts:
  cuda_visible_devices: '0'
omni:
  cuda_visible_devices: '1'
  max_num_seqs: 8
  gpu_memory_utilization: 0.78
  kv_cache_memory_bytes: 2147483648
vlm:
  cuda_visible_devices: '0'
  gpu_memory_utilization: 0.47
  kv_cache_memory_bytes: 1610612736
clef: {}

model-servers/yaml/spark.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Other settings come from each service's YAML under services/.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 4
  gpu_memory_utilization: 0.25
  kv_cache_memory_bytes: 2147483648
  spark_uma: true
vlm:
  gpu_memory_utilization: 0.2
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4
  spark_uma: true
clef: {}

yaml/conversation_example_worker.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Conversation-example worker configuration.
# Auto-discovered by the launcher and passed as --config to the worker process.

# Path to the voice-gate config (resolved relative to this file's
# directory). Owns the magic-phrase / follow-up / listening-chime knobs.
voice_gate_yaml: voice_gate.yaml

# The worker uses its packaged prompts/system.txt by default. Set
# system_prompt or system_prompt_file here only to override it.

# Frame freshness — tune if the agent sees stale frames.
frame_max_age_s: 5.0   # frame older than this is considered stale
frame_timeout_s: 5.0   # give up waiting for a fresh frame after this many seconds

# Voice Activity Detection — Silero VAD (ONNX backend).
# Tune if STT triggers too early or misses speech.
silero_threshold:  0.5    # Silero speech-probability gate (0..1); lower = more sensitive
silence_duration:  0.4    # seconds of silence that ends an utterance
min_speech:        0.1    # minimum seconds of speech before STT is called

# ── Idle timeout ──────────────────────────────────────────────────────────
# Auto-cancel the voice pipeline after this many seconds with no user/bot
# speech. DISABLED by default (0 = off) — a quiet session stays connected
# indefinitely, which is what XR sessions usually want. Set a positive value
# (e.g. 300 for 5 min) to opt in; the worker then tears the pipeline down
# after that long idle. Applied by VoiceAgent.
idle_timeout_secs: 0

# General-conversation context budget, split evenly between completed dialogue
# and saved notes. Each keeps recent entries verbatim and summarizes older ones.
context_budget_bytes: 12000
# Completed history the router sends with each routing decision. Clef accepts
# at most 4,096 tokens per request, including the routing prompt and destinations.
routing_history_budget_bytes: 8000
# Input limit for one asynchronous context compaction request.
compaction_input_budget_bytes: 12000

# Optional route-aware intention acknowledgment. When omitted, prediction and
# acknowledgment are disabled. Set a positive delay in seconds to enable it.
# The selected destination starts immediately; a confident slow prediction can
# speak an acknowledgment early, otherwise the delay is the fallback threshold.
# acknowledgment_delay_secs: 2.0

# Background visual alerts pace their frame checks. Every frame is first screened
# by a cheap Clef image call; only a frame that may show the alert's subject also
# runs Cosmos and the condition check. A screened frame waits at least
# alert_confirm_interval_secs before the next check, otherwise
# alert_check_interval_secs (0 <= check <= confirm).
# alert_check_interval_secs: 1.0
# alert_confirm_interval_secs: 3.0

yaml/device_io_hub.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# DeviceIOHub configuration for the conversation-example sample.
# Paths are relative to this file's directory.
# See services/device-io-hub/device_io_hub.yaml for all available options.

# ── LiveKit server ────────────────────────────────────────────────────────────
# Leave both blank to generate credentials for this hub run.
# Optional overrides: LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the environment.
# Refer to docs/source/getting_started/credentials.md for setup.
api_key:    ""
api_secret: ""
room_name:  xr-room

lk_port_ws:  7880
lk_port_tcp: 7881
lk_port_udp: 7882

# Cloud VMs behind NAT may need STUN-based public-IP discovery. Also skip
# validation when the NAT does not support LiveKit's public-IP self-ping.
# lk_use_external_ip: true
# lk_skip_external_ip_validation: true

# ── Web client server ─────────────────────────────────────────────────────────
enable_web_server: true
web_server_port:   8080
web_client_dir:    ../../../client-samples/web

# ── Optional: token server ────────────────────────────────────────────────────
enable_token_server: true
# Change if another service holds port 8000 (e.g. iphlpsvc under WSL2 mirrored
# networking; see the note in services/device-io-hub/device_io_hub.yaml).
# token_server_port: 8000

yaml/media_capture.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Other session recording settings use the shared defaults. Add fields here to override them.
out_dir: ~/.local/share/xr-ai/captures/conversation-example
profile: demo

yaml/models.json#

{
  "models": {
    "vlm": {
      "adapter": {"preset": "cosmos3_nano_reasoner"},
      "endpoint": {"base_url": "http://localhost:8100"}
    },
    "llm": {
      "adapter": {
        "preset": "nemotron_omni",
        "thinking": {"control": "chat_template", "default": true},
        "default_extras": {
          "max_tokens": 4608,
          "temperature": 0.25,
          "chat_template_kwargs": {
            "reasoning_budget": 4096
          }
        }
      },
      "endpoint": {"base_url": "http://localhost:8108"}
    },
    "router": {
      "category": "decision",
      "kind": "systemone",
      "model_name": "Cloudflare/clef-flash",
      "base_url": "http://localhost:8120"
    },
    "tts": {
      "adapter": {"preset": "pocket_tts"},
      "endpoint": {"base_url": "http://localhost:8105"}
    }
  }
}

yaml/voice_gate.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Voice-gate (speech-input opt-in) configuration for conversation-example.
# Consumed by xr_ai_voicegate.VoiceGate inside the unified pipecat pipeline.
#
# Magic phrases are sentence-boundary opt-ins: an STT transcript must begin
# with one of these or contain one after '.', '?', or '!' followed by
# whitespace or a closing quote. Matching is case-insensitive. Text through
# the matched phrase is stripped before the agent sees the query; commas and
# other mid-sentence mentions do not match. The same stripped tail is used for
# early global-STOP detection; the complete final transcript still determines
# whether a stop-like opening is a global stop or a scoped agent command.
mode: conversation
magic_phrases:
  - "agent"
  - "hey agent"

# Audible "I heard you" feedback. When true (and magic_phrases is
# non-empty), a short two-tone chime plays the moment a phrase matches —
# before STT+VLM finish — so the user knows the agent is processing.
listening_chime: false

# Enable global STOP interruption and final-transcript acknowledgement.
# Set false for passive narration; probes and conversation controls stay independent.
# Scoped commands such as "stop recording" remain ordinary agent queries.
stop_commands_enabled: true

# Follow-up grace window. After a bare magic phrase without a question, the
# next utterance from the same participant within this many seconds
# bypasses the gate so "hey agent" → pause → "what am I looking at?"
# works as one natural interaction. Set to 0 to disable.
followup_grace_s: 5.0

# Start once, then speak naturally until the full stop phrase.
# A short "stop" interrupts the response without closing the conversation.
conversation:
  start_phrase: "Hey agent, let's start talking"
  stop_phrase: "Hey agent, let's stop talking"
  # false: speak naturally after starting. true: keep wake and follow-up rules.
  require_wake_phrase: false
  # Join exact control fragments whose onsets are within this window.
  phrase_window_s: 6.0

# The default stack supplies speaker ASR; wait for it before worker readiness.
# For explicit batch STT, set speaker.enabled false and add an stt model entry.
speaker:
  enabled: true
  backend: required
  base_url: http://127.0.0.1:8102

lab-instrument-monitoring#

model-servers/yaml/96G_blackwell.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 8
  gpu_memory_utilization: 0.35
  kv_cache_memory_bytes: 2147483648
vlm:
  gpu_memory_utilization: 0.23
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4

model-servers/yaml/dual_48G_ada.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '1'
tts:
  cuda_visible_devices: '0'
omni:
  cuda_visible_devices: '1'
  max_num_seqs: 8
  gpu_memory_utilization: 0.78
  kv_cache_memory_bytes: 2147483648
vlm:
  cuda_visible_devices: '0'
  gpu_memory_utilization: 0.47
  kv_cache_memory_bytes: 1610612736

model-servers/yaml/spark.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 4
  gpu_memory_utilization: 0.25
  kv_cache_memory_bytes: 2147483648
  spark_uma: true
vlm:
  gpu_memory_utilization: 0.2
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4
  spark_uma: true

yaml/device_io_hub.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Leave both blank to generate credentials for this hub run.
# Optional overrides: LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the environment.
# Refer to docs/source/getting_started/credentials.md for setup.
api_key: ""
api_secret: ""
room_name: xr-room

lk_port_ws: 7880
lk_port_tcp: 7881
lk_port_udp: 7882

enable_web_server: true
web_server_port: 8080
web_client_dir: ../../../client-samples/web
enable_token_server: true

yaml/device_map.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

devices:
  qr_code:
    device-1: Device1
    device-2: Device2
    device-3: Device3
    device-4: Device4
    device-5: Device5
  aruco:
    "0": Device1
    "1": Device2
    "2": Device3
    "3": Device4
    "4": Device5

yaml/lab_instrument_monitoring_worker.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

voice_gate_yaml: voice_gate.yaml
device_map_yaml: device_map.yaml
artifacts_dir: ../artifacts
capture_marker_scans: false
web_events_host: 127.0.0.1
web_events_port: 8092
web_events_max_events: 5000

monitor_interval_s: 5.0
instrument_monitor_interval_s: 5.0
instrument_state_interval_s: 10.0
instrument_lost_after_s: 30.0
monitor_history_size: 20
frame_max_age_s: 5.0
frame_timeout_s: 5.0

silero_threshold: 0.5
silence_duration: 0.4
min_speech: 0.1
idle_timeout_secs: 0

yaml/media_capture.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Other session recording settings use the shared defaults. Add fields here to override them.
out_dir: ~/.local/share/xr-ai/captures/lab-instrument-monitoring
profile: demo

yaml/models.json#

{
  "models": {
    "llm": {
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    },
    "vlm": {
      "adapter": {"preset": "cosmos3_nano_reasoner"},
      "endpoint": {"base_url": "http://localhost:8100"}
    },
    "tts": {
      "adapter": {"preset": "pocket_tts"},
      "endpoint": {"base_url": "http://localhost:8105"}
    }
  }
}

yaml/voice_gate.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Final transcripts during an active conversation are recorded before any
# optional wake gate. Only accepted turns reach the foreground agent.
mode: conversation
magic_phrases:
  - agent
  - hey agent
listening_chime: true
followup_grace_s: 5.0

# Start once, then speak naturally until the full stop phrase.
conversation:
  start_phrase: "Hey agent, let's start talking"
  stop_phrase: "Hey agent, let's stop talking"
  require_wake_phrase: false  # true keeps magic_phrases and followup_grace_s
  phrase_window_s: 6.0

# The default stack supplies speaker ASR; wait for it before worker readiness.
# For explicit batch STT, set speaker.enabled false and add an stt model entry.
speaker:
  enabled: true
  backend: required
  base_url: http://127.0.0.1:8102

procedure-capture-guide#

model-servers/yaml/96G_blackwell.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Explicit KV bytes reserve the captioning cache; utilization controls admission.
stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 8
  gpu_memory_utilization: 0.35
  kv_cache_memory_bytes: 2147483648
vlm:
  gpu_memory_utilization: 0.23
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4

model-servers/yaml/dual_48G_ada.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Separate STT from the VLM; explicit KV bytes reserve the captioning cache.
stt:
  cuda_visible_devices: '1'
tts:
  cuda_visible_devices: '0'
omni:
  cuda_visible_devices: '1'
  max_num_seqs: 8
  gpu_memory_utilization: 0.78
  kv_cache_memory_bytes: 2147483648
vlm:
  cuda_visible_devices: '0'
  gpu_memory_utilization: 0.47
  kv_cache_memory_bytes: 1610612736

model-servers/yaml/spark.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Explicit KV bytes reserve the captioning cache; utilization controls admission.
stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 4
  gpu_memory_utilization: 0.25
  kv_cache_memory_bytes: 2147483648
  spark_uma: true
vlm:
  gpu_memory_utilization: 0.2
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4
  spark_uma: true

yaml/capture-agent-config.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Capture policy; voice and frame infrastructure settings live in worker.yaml.
# Paths are relative to this YAML. --output-root overrides artifacts_dir.
# Packets are written under artifacts_dir/sessions/.
artifacts_dir: ../artifacts
capture_fps: 2.0
caption_interval_s: 5.0

yaml/device_io_hub.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# DeviceIOHub configuration for the procedure-capture-guide sample.
# Paths are relative to this file's directory.
# See services/device-io-hub/device_io_hub.yaml for all available options.

# ── LiveKit server ────────────────────────────────────────────────────────────
# Leave both blank to generate credentials for this hub run.
# Optional overrides: LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the environment.
# Refer to docs/source/getting_started/credentials.md for setup.
api_key:    ""
api_secret: ""
room_name:  xr-room

lk_port_ws:  7880
lk_port_tcp: 7881
lk_port_udp: 7882

# Cloud VMs behind NAT may need STUN-based public-IP discovery. Also skip
# validation when the NAT does not support LiveKit's public-IP self-ping.
# lk_use_external_ip: true
# lk_skip_external_ip_validation: true

# ── Web client server ─────────────────────────────────────────────────────────
enable_web_server: true
web_server_port:   8080
web_client_dir:    ../../../client-samples/web

# ── Optional: token server ────────────────────────────────────────────────────
enable_token_server: true
# Change if another service holds port 8000 (e.g. iphlpsvc under WSL2 mirrored
# networking; see the note in services/device-io-hub/device_io_hub.yaml).
# token_server_port: 8000

yaml/guide-agent-config.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Guide-only verification settings. The CLI selects the guide file; models are shared with capture.
vision_timeout_s: 15.0

yaml/media_capture.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Session recording covers the whole participant connection, independently of
# spoken capture or guide commands in either mode. Other settings use shared defaults.
out_dir: ~/.local/share/xr-ai/captures/procedure-capture-guide
profile: demo
session_mode: participant

yaml/models.json#

{
  "models": {
    "llm": {
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    },
    "stt": {
      "adapter": {"preset": "parakeet_stt"},
      "endpoint": {"base_url": "http://localhost:8103"}
    },
    "vlm": {
      "adapter": {"preset": "cosmos3_nano_reasoner"},
      "endpoint": {"base_url": "http://localhost:8100"}
    },
    "tts": {
      "adapter": {"preset": "pocket_tts"},
      "endpoint": {"base_url": "http://localhost:8105"}
    }
  }
}

yaml/voice_gate.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Shared wake-word-free input for capture and guide modes. Pass final speech
# to the selected agent so it can handle its own start and stop commands;
# generic STOP handling must not consume those commands or narration.
magic_phrases: []
listening_chime: false
stop_commands_enabled: false
followup_grace_s: 0.0

yaml/worker.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Shared frame and voice settings. Configuration paths are relative to this YAML.
capture_config: capture-agent-config.yaml
guide_config: guide-agent-config.yaml
frame_max_age_s: 2.0
frame_timeout_s: 3.0
silence_duration: 0.6
min_speech: 0.15
silero_threshold: 0.4

simple-vlm-example#

model-servers/yaml/96G_blackwell.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
vlm:
  gpu_memory_utilization: 0.23
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4

model-servers/yaml/dual_48G_ada.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '1'
tts:
  cuda_visible_devices: '0'
vlm:
  cuda_visible_devices: '0'
  gpu_memory_utilization: 0.47
  kv_cache_memory_bytes: 1610612736

model-servers/yaml/spark.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
vlm:
  gpu_memory_utilization: 0.2
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4
  spark_uma: true

yaml/device_io_hub.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# DeviceIOHub configuration for the simple-vlm-example sample.
# Paths are relative to this file's directory.
# See services/device-io-hub/device_io_hub.yaml for all available options.

# ── LiveKit server ────────────────────────────────────────────────────────────
# Leave both blank to generate credentials for this hub run.
# Optional overrides: LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the environment.
# Refer to docs/source/getting_started/credentials.md for setup.
api_key:    ""
api_secret: ""
room_name:  xr-room

lk_port_ws:  7880
lk_port_tcp: 7881
lk_port_udp: 7882

# An omitted lk_internal_url follows lk_port_ws; explicit URLs are preserved.

# Cloud VMs behind NAT may need STUN-based public-IP discovery. Also skip
# validation when the NAT does not support LiveKit's public-IP self-ping.
# lk_use_external_ip: true
# lk_skip_external_ip_validation: true

# ── Web client server ─────────────────────────────────────────────────────────
enable_web_server: true
web_server_port:   8080
web_client_dir:    ../../../client-samples/web

# ── Optional: token server ────────────────────────────────────────────────────
enable_token_server: true
# Change if another service holds port 8000 (e.g. iphlpsvc under WSL2 mirrored
# networking; see the note in services/device-io-hub/device_io_hub.yaml).
# token_server_port: 8000
# An omitted token_server_url follows token_server_port; explicit URLs are preserved.

yaml/media_capture.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Other session recording settings use the shared defaults. Add fields here to override them.
out_dir: ~/.local/share/xr-ai/captures/simple-vlm-example
profile: demo

yaml/models.json#

{
  "models": {
    "vlm": {
      "adapter": {"preset": "cosmos3_nano_reasoner"},
      "endpoint": {"base_url": "http://localhost:8100"}
    },
    "tts": {
      "adapter": {"preset": "pocket_tts"},
      "endpoint": {"base_url": "http://localhost:8105"}
    }
  }
}

yaml/simple_vlm_example_worker.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Simple VLM example worker configuration.
# Auto-discovered by the launcher and passed as --config to the worker process.

# Path to the voice-gate config (resolved relative to this file's
# directory). Owns the magic-phrase / follow-up / listening-chime knobs.
voice_gate_yaml: voice_gate.yaml

# The worker uses its packaged prompts/system.txt by default. Set
# system_prompt or system_prompt_file here only to override it.

# Frame freshness — tune if the agent sees stale frames.
frame_max_age_s: 5.0   # frame older than this is considered stale
frame_timeout_s: 5.0   # give up waiting for a fresh frame after this many seconds

# Voice Activity Detection — Silero VAD (ONNX backend).
# Tune if STT triggers too early or misses speech.
silero_threshold:  0.5    # Silero speech-probability gate (0..1); lower = more sensitive
silence_duration:  0.4    # seconds of silence that ends an utterance
min_speech:        0.1    # minimum seconds of speech before STT is called

# ── Idle timeout ──────────────────────────────────────────────────────────
# Auto-cancel the voice pipeline after this many seconds with no user/bot
# speech. DISABLED by default (0 = off) — a quiet session stays connected
# indefinitely, which is what XR sessions usually want. Set a positive value
# (e.g. 300 for 5 min) to opt in; the worker then tears the pipeline down
# after that long idle. Applied by VoiceAgent.
idle_timeout_secs: 0

yaml/voice_gate.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Voice-gate (speech-input opt-in) configuration for simple-vlm-example.
# Consumed by xr_ai_voicegate.VoiceGate inside the unified pipecat pipeline.
#
# Magic phrases are sentence-boundary opt-ins: an STT transcript must begin
# with one of these or contain one after '.', '?', or '!' followed by
# whitespace or a closing quote. Matching is case-insensitive. Text through
# the matched phrase is stripped before the agent sees the query; commas and
# other mid-sentence mentions do not match. The same stripped tail is used for
# early global-STOP detection; the complete final transcript still determines
# whether a stop-like opening is a global stop or a scoped agent command.
mode: conversation
magic_phrases:
  - "agent"
  - "hey agent"

# Audible "I heard you" feedback. When true (and magic_phrases is
# non-empty), a short two-tone chime plays the moment a phrase matches —
# before STT+VLM finish — so the user knows the agent is processing.
listening_chime: false

# Enable global STOP interruption and final-transcript acknowledgement.
# Set false for passive narration; probes and conversation controls stay independent.
# Scoped commands such as "stop recording" remain ordinary agent queries.
stop_commands_enabled: true

# Follow-up grace window. After a bare magic phrase without a question, the
# next utterance from the same participant within this many seconds
# bypasses the gate so "hey agent" → pause → "what am I looking at?"
# works as one natural interaction. Set to 0 to disable.
followup_grace_s: 5.0

# Start once, then speak naturally until the full stop phrase.
# A short "stop" interrupts the response without closing the conversation.
conversation:
  start_phrase: "Hey agent, let's start talking"
  stop_phrase: "Hey agent, let's stop talking"
  # false: speak naturally after starting. true: keep wake and follow-up rules.
  require_wake_phrase: false
  # Join exact control fragments whose onsets are within this window.
  phrase_window_s: 6.0

# The default stack supplies speaker ASR; wait for it before worker readiness.
# For explicit batch STT, set speaker.enabled false and add an stt model entry.
speaker:
  enabled: true
  backend: required
  base_url: http://127.0.0.1:8102
  # Display detected/ignored speakers in RECEIVED; adds ASR work for other voices.
  diagnostics: true

tea-making-sample#

model-servers/yaml/96G_blackwell.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 8
  gpu_memory_utilization: 0.35
  kv_cache_memory_bytes: 2147483648
embedding:
  gpu_memory_utilization: 0.08

model-servers/yaml/dual_48G_ada.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '1'
tts:
  cuda_visible_devices: '0'
omni:
  cuda_visible_devices: '1'
  max_num_seqs: 8
  gpu_memory_utilization: 0.78
  kv_cache_memory_bytes: 2147483648
embedding:
  cuda_visible_devices: '0'
  gpu_memory_utilization: 0.08

model-servers/yaml/spark.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 4
  gpu_memory_utilization: 0.25
  kv_cache_memory_bytes: 2147483648
  spark_uma: true
embedding:
  gpu_memory_utilization: 0.08
  spark_uma: true
  # NGC 26.09 had incompatible FlashAttention kernels on GB10 SM121/driver 580.
  # Retain Triton pending NGC 26.10 preview qualification on this profile.
  attention_backend: TRITON_ATTN

yaml/device_io_hub.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Leave both blank to generate credentials for this hub run.
# Optional overrides: LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the environment.
# Refer to docs/source/getting_started/credentials.md for setup.
api_key: ""
api_secret: ""
room_name: xr-room

lk_port_ws: 7880
lk_port_tcp: 7881
lk_port_udp: 7882

enable_web_server: true
web_server_port: 8080
web_client_dir: ../../../client-samples/web
enable_token_server: true

yaml/media_capture.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Other session recording settings use the shared defaults. Add fields here to override them.
out_dir: ~/.local/share/xr-ai/captures/tea-making-sample
profile: demo

yaml/models.local.json#

{
  "models": {
    "llm": {
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    },
    "vlm": {
      "category": "vlm",
      "adapter": {
        "kind": "openai_compat",
        "model_name": "llm",
        "reasoning_field": "reasoning_content",
        "capabilities": {"streaming": true, "vision": true},
        "default_extras": {
          "max_tokens": 128,
          "temperature": 0.0,
          "chat_template_kwargs": {"enable_thinking": false}
        }
      },
      "endpoint": {"base_url": "http://localhost:8108"}
    },
    "embedding": {
      "adapter": {"preset": "nemotron_embedding"},
      "endpoint": {"base_url": "http://localhost:8109"}
    },
    "tts": {
      "adapter": {"preset": "pocket_tts"},
      "endpoint": {"base_url": "http://localhost:8105"}
    }
  }
}

yaml/rag_service.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

endpoint: tcp://0.0.0.0:8340
documents_dir: ../rag-documents
models_config: models.local.json
embedding_role: embedding
cache_dir: ../../../models/tea-making-rag-cache
chunk_size: 700
overlap: 100
# Native output dimension of both Nemotron 3 Embed 1B BF16 and its NIM.
embedding_dim: 2048
batch_size: 16
min_score: 0.3

yaml/tea_making_worker.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# The launcher materializes these paths for runtime path resolution.
models_config: models.local.json
workflow_config: workflow.yaml
voice_gate_yaml: voice_gate.yaml

# Choose one to replace the idle and explicit background-action instructions.
# Active tea guidance comes from the focused step policies in workflow.yaml.
# foreground_prompt: <inline shared foreground instructions>
# foreground_prompt_file: ../worker/tea_making_worker/prompts/foreground_prompt.txt

rag_endpoint: tcp://127.0.0.1:8340
artifacts_dir: ../artifacts

web_events_host: 127.0.0.1
web_events_port: 8092
web_events_max_events: 5000

frame_max_age_s: 3.0
frame_timeout_s: 5.0
vlm_timeout_s: 15.0

change_watch_interval_s: 2.0
transcript_summary_interval_s: 120.0
video_log_interval_s: 2.0
background_history_size: 50

change_watch_default_instruction: >-
  important changes involving people, objects, actions, or possible hazards

silero_threshold: 0.3
silence_duration: 1.2
min_speech: 0.15
idle_timeout_secs: 0

yaml/voice_gate.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

mode: conversation
magic_phrases:
  - agent
  - hey agent
listening_chime: true
followup_grace_s: 5.0

# Start once, then speak naturally until the full stop phrase.
conversation:
  start_phrase: "Hey agent, let's start talking"
  stop_phrase: "Hey agent, let's stop talking"
  require_wake_phrase: false  # true keeps magic_phrases and followup_grace_s
  phrase_window_s: 6.0

# The default stack supplies speaker ASR; wait for it before worker readiness.
# For explicit batch STT, set speaker.enabled false and add an stt model entry.
speaker:
  enabled: true
  backend: required
  base_url: http://127.0.0.1:8102

yaml/workflow.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

task:
  name: tea making
  start_step: identify
  foreground_prompt: This workflow guides tea making.
  complete_message: Tea guidance complete. Enjoy your tea.

state:
  tea_name:
    type: string
    description: Tea name from the current caption.
  target_temperature_c:
    type: number
    description: Brewing temperature in Celsius.
  steep_duration_s:
    type: integer
    description: Steep duration in seconds.
  guidance_source:
    type: string
    description: Package or retrieved source.
  tea_ready:
    type: boolean
    description: True only when all preceding fields are supported.
  water_filled:
    type: boolean
    description: Whether water is visibly in the heating vessel.
    initial: false
  heating_started:
    type: boolean
    description: Whether a heater-on indicator or unit-bearing temperature was seen.
    initial: false
  steeping_started_at_us:
    type: integer
    description: Start time returned by clock__now.
  steeping_started:
    type: boolean
    description: Whether tea is immersed and its start time is recorded.
    initial: false
  steeping_complete:
    type: boolean
    description: Whether the fresh timer result is expired.
    initial: false

steps:
  - id: identify
    title: Identify the tea
    reads: []
    writes: [tea_name, target_temperature_c, steep_duration_s, guidance_source, tea_ready]
    trigger:
      function: current_view
      interval_s: 1.5
      arguments:
        question: >-
          Read the front label brand and tea/blend name, plus visible brewing
          temperature/time. Ignore slogans, claims, bag count, weight, and
          badges. Transcribe exactly; never infer. If no brand or tea/blend is
          legible, say label unreadable.
      result_field: answer
    evidence:
      pattern: "^accepted$"
      consecutive: 1
    agent:
      prompt: >-
        Identify tea only from the current caption, never state or RAG.
        If the tea name is unreadable, commit empty updates; do not retrieve or
        write placeholder values. Prefer
        visible values. If temperature/time is missing,
        call rag_lookup once with "<exact name> water temperature steep time",
        top_k 2; accept only a section naming it. If every writable field is
        supported, commit all with tea_ready true. Otherwise commit nothing;
        never write tea_ready false.
      tools: [rag_lookup]
    voice:
      prompt: >-
        Tea identity comes only from state tea_name or current_view, never RAG.
        tea_ready means the tea instructions were identified, not that the
        drink has been prepared.
        If the exact name is unknown, inspect the current frame. Retrieve only
        missing temperature or duration after the exact name is known. Never
        replace package values. Answer only what was asked.
      tools: [current_view, rag_lookup]
    complete_when: {tea_ready: true}
    next: fill_water
    state_on_skip:
      tea_name: generic tea
      target_temperature_c: 93
      steep_duration_s: 180
      guidance_source: generic fallback
      tea_ready: true
    messages:
      enter: Hold the tea package or tag in view so I can read it.
      complete: I found the tea instructions. Say next when ready.
      skip: I will use generic tea defaults.

  - id: fill_water
    title: Fill the heating vessel
    reads: [tea_name, target_temperature_c, steep_duration_s]
    writes: [water_filled]
    trigger:
      function: current_view
      interval_s: 2
      arguments:
        question: >-
          Describe the visible interior and contents of the kettle or pot.
          State whether water is visible inside and what supports that reading.
          If the contents are unclear, say so. Never infer.
      result_field: answer
    evidence:
      pattern: "^accepted$"
      consecutive: 2
    agent:
      prompt: >-
        Judge the fresh caption semantically without requiring particular
        words. Set water_filled true when it reasonably shows water inside the
        vessel. Otherwise commit empty updates and message.
      tools: []
    voice:
      prompt: >-
        Give instructions or verify with current_view. A user report is not
        visual confirmation. Confirm filled only from state water_filled or a
        current view showing the vessel interior and concrete water cues;
        otherwise say it is not yet visually confirmed.
      tools: [current_view]
    complete_when: {water_filled: true}
    next: heat_water
    state_on_skip: {water_filled: true}
    messages:
      enter: Fill a kettle or pot with fresh water and keep it in view.
      complete: The vessel has water. Say next when ready.
      skip: I will assume the vessel is filled.

  - id: heat_water
    title: Heat the water
    reads: [tea_name, target_temperature_c, steep_duration_s]
    writes: [heating_started]
    trigger:
      function: current_view
      interval_s: 2
      arguments:
        question: >-
          Describe the visible kettle or thermometer display and transcribe its
          current numeric reading and unit. If either is unclear, say so. Do not
          confuse a printed target temperature with the current reading.
      result_field: answer
    evidence:
      pattern: "^accepted$"
      consecutive: 2
    agent:
      prompt: >-
        This step detects that heating is underway; it does not wait for the tea
        target temperature. Ignore target_temperature_c when deciding this
        step. For every fresh current reading with an explicit Celsius or
        Fahrenheit unit, first call temperature__threshold with that exact
        reading and a threshold_c of 50. If its above result is true, commit
        heating_started true. If above is false, commit empty updates and
        message with rejected evidence. For a target value, ambiguous unit,
        missing reading, or unclear current-versus-target value, do not call the
        tool; commit empty updates and message with unknown evidence so it does
        not erase a prior confirmed reading.
      tools: [temperature__threshold]
    voice:
      prompt: >-
        Explain target from state. For a reading, call current_view and
        repeat the visible number and unit; do not verify. For hot-enough
        checks, call current_view, then temperature__verify only with an exact
        number and Celsius or Fahrenheit unit. Report result. If missing,
        say the reading is unavailable.
      tools: [current_view, temperature__verify]
    complete_when: {heating_started: true}
    next: start_steeping
    state_on_skip: {heating_started: true}
    messages:
      enter: Heat the water to {{ target_temperature_c | temperature_c }}.
      complete: Heating is underway. Wait for the water to reach the target temperature.
      skip: I will assume heating has started.

  - id: start_steeping
    title: Start steeping
    reads: [tea_name, target_temperature_c, steep_duration_s]
    writes: [steeping_started_at_us, steeping_started]
    trigger:
      function: current_view
      interval_s: 2
      arguments:
        question: >-
          Look for two separately visible facts in one vessel: a water surface
          or level; and a tea bag, infuser, or leaves entering or immersed in
          that water. If either water or tea-water contact is not visible, say
          it is unconfirmed. Note hazards.
      result_field: answer
    evidence:
      pattern: "^accepted$"
      consecutive: 2
    agent:
      prompt: >-
        Judge when steeping starts, not when it finishes. Treat unconfirmed,
        unknown, unclear, or missing visual evidence as a hard stop: do not call
        clock__now, and commit empty updates and message with unknown evidence.
        Only when the fresh caption explicitly confirms both visible liquid and
        tea visibly contacting that liquid inside the same vessel, first call
        clock__now. Nearby, above, or outside the liquid does not count. Then
        commit exactly its epoch_us as steeping_started_at_us with
        steeping_started true. For any other caption, do not infer contact from
        context or prior state; commit empty updates and message. Never invent or
        replace a start time.
      tools: [clock__now]
    voice:
      prompt: >-
        Give instructions or verify with current_view. A report or tea item in
        a vessel is not confirmation. Require both visible water and visible
        tea-water contact. The observation loop, not this answer, starts the
        timer.
      tools: [current_view]
    complete_when: {steeping_started: true}
    next: steep_timer
    complete_on_skip: true
    state_on_skip: {steeping_started: true}
    messages:
      enter: Put the tea in the water; I will start the timer when I see it immersed.
      complete: Steeping timer started. Say next to monitor it.
      skip: Steeping detection and the timer were skipped.

  - id: steep_timer
    title: Wait for steeping
    reads:
      [tea_name, target_temperature_c, steeping_started_at_us, steep_duration_s]
    writes: [steeping_complete]
    trigger:
      function: clock__timer
      interval_s: 1
      arguments:
        started_at_us: $state.steeping_started_at_us
        duration_s: $state.steep_duration_s
    agent:
      prompt: >-
        Copy the fresh clock__timer expired boolean: set steeping_complete true
        only when expired is true. While false, commit empty updates/message.
        Never infer from elapsed text, wall time, or prior state.
      tools: []
    voice:
      prompt: >-
        Call clock__timer for every timer or completion question. Report only
        its fresh remaining time or completion; never calculate from state,
        prior answers, or wall-clock time.
      tools: [clock__timer]
    complete_when: {steeping_complete: true}
    next: null
    state_on_skip: {steeping_complete: true}
    messages:
      enter: Steep for {{ steep_duration_s | duration }}.
      complete: Steeping is complete. Remove the tea, then say next to finish.
      skip: Timer skipped; remove the tea when it tastes right.

xr-render-demo#

model-servers/yaml/96G_blackwell.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 8
  gpu_memory_utilization: 0.35
  kv_cache_memory_bytes: 2147483648
vlm:
  gpu_memory_utilization: 0.23
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4

model-servers/yaml/dual_48G_ada.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '1'
tts:
  cuda_visible_devices: '0'
omni:
  cuda_visible_devices: '1'
  max_num_seqs: 8
  gpu_memory_utilization: 0.78
  kv_cache_memory_bytes: 2147483648
vlm:
  cuda_visible_devices: '0'
  gpu_memory_utilization: 0.47
  kv_cache_memory_bytes: 1610612736

model-servers/yaml/spark.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample workload overrides. Ports, models, and images come from each service's own YAML.
# These retain the existing conservative budgets; tune together on the target GPU.
# With explicit KV bytes, utilization is startup admission, not a total-memory cap.
speaker-stt:
  cuda_visible_devices: '0'
tts:
  cuda_visible_devices: '0'
omni:
  max_num_seqs: 4
  gpu_memory_utilization: 0.25
  kv_cache_memory_bytes: 2147483648
  spark_uma: true
vlm:
  gpu_memory_utilization: 0.2
  kv_cache_memory_bytes: 1610612736
  max_num_seqs: 4
  spark_uma: true

scene/scene_service.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Sample-local scene process reference configuration.
# LOVR_BIN in the environment takes priority over lovr_bin.
# lovr_bin: /home/you/hub/lovr/build/bin/lovr

xr_app_dir: ./lovr
endpoint: tcp://0.0.0.0:8320
scene_socket: ipc:///tmp/xr_render_scene
cloudxr_env_file: ~/.cloudxr/run/cloudxr.env

yaml/cloudxr_runtime.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# CloudXR runtime configuration.
# Full docs: python -m isaacteleop.cloudxr --help

# Directory where CloudXR stores runtime state, certs, and logs (default: ~/.cloudxr).
cloudxr_install_dir: ~/.cloudxr

# Set to true to accept the NVIDIA CloudXR EULA non-interactively.
# EULA: https://github.com/NVIDIA/IsaacTeleop/blob/main/deps/cloudxr/CLOUDXR_LICENSE
# Once accepted the marker is written to <cloudxr_install_dir>/run/eula_accepted
# and this flag is no longer checked.
accept_eula: true

# CloudXR env overrides applied before the native service starts.
# NV_DEVICE_PROFILE selects the client type:
#   auto-webrtc: WebRTC / web XR clients (default)
#   auto-native: native iOS / visionOS clients (set this to use the native apps)
# Device-specific values also accepted: apple-vision-pro | ipad-pro | quest3
cloudxr_env:
  NV_DEVICE_PROFILE: auto-webrtc

# Physical GPU (nvidia-smi index) the CloudXR compositor pins to. Translated
# to PCI bus selectors for Vulkan, CUDA, and Mesa, set on this process's
# environment, and appended to cloudxr.env so the scene process picks them up
# when it sources the file (and LOVR inherits from the scene spawn env).
gpu_index: 0

# ── Network ports (CloudXR owns these — do not conflict with LiveKit) ─────────
# CloudXR native service:  localhost:49100  (internal, not exposed)
# WSS proxy (TLS):         0.0.0.0:48322   (XR clients connect here)

yaml/device_io_hub.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# DeviceIOHub configuration for the xr-render-demo sample.
# Paths are relative to this file's directory.
# See services/device-io-hub/device_io_hub.yaml for all available options.

# ── LiveKit server ────────────────────────────────────────────────────────────
# Leave both blank to generate credentials for this hub run.
# Optional overrides: LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the environment.
# Refer to docs/source/getting_started/credentials.md for setup.
api_key:    ""
api_secret: ""
room_name:  xr-room

lk_port_ws:  7880
lk_port_tcp: 7881
lk_port_udp: 7882

# Cloud VMs behind NAT may need STUN-based public-IP discovery. Also skip
# validation when the NAT does not support LiveKit's public-IP self-ping.
# lk_use_external_ip: true
# lk_skip_external_ip_validation: true

# ── Web client server ─────────────────────────────────────────────────────────
# The page serves both stacks: LiveKit (agent audio/data over the WSS proxy on
# this port) AND CloudXR (one-tap Launch XR → WebXR over wss://<host>:48322).
# Because both live in the same JS context, entering XR does not drop the
# agent connection.
enable_web_server: true
web_server_port:   8080
web_client_dir:    ../../../client-samples/web-xr

# ── Optional: token server ────────────────────────────────────────────────────
enable_token_server: true
# Change if another service holds port 8000 (e.g. iphlpsvc under WSL2 mirrored
# networking; see the note in services/device-io-hub/device_io_hub.yaml).
# token_server_port: 8000

# ── Video recording ──────────────────────────────────────────────────────────
# Disabled: xr-render-demo obtains current frames through LiveFrameSource.
# Enable this and set recordings_dir in video_memory_service.yaml for history.
video_recording:
  enabled: false

yaml/media_capture.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Other session recording settings use the shared defaults. Add fields here to override them.
out_dir: ~/.local/share/xr-ai/captures/xr-render-demo
profile: demo

yaml/models.json#

{
  "models": {
    "llm": {
      "adapter": {"preset": "nemotron_omni"},
      "endpoint": {"base_url": "http://localhost:8108"}
    },
    "tts": {
      "adapter": {"preset": "pocket_tts"},
      "endpoint": {"base_url": "http://localhost:8105"}
    },
    "vlm": {
      "adapter": {"preset": "cosmos3_nano_reasoner"},
      "endpoint": {"base_url": "http://localhost:8100"}
    }
  }
}

yaml/openxr_service.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

endpoint: tcp://0.0.0.0:8330
cloudxr_env_file: ~/.cloudxr/run/cloudxr.env

# Test hook: the eval live drivers need a simulated head pose; flip to true
# while running them. The injected pose overrides real tracking for every
# client until cleared, so it must never ship enabled.
allow_sim_pose: false

yaml/video_memory_service.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

endpoint: tcp://0.0.0.0:8310
out_dir: /tmp/xr_video_queries/xr-render-demo
# Must match device_io_hub.yaml video_recording.out_dir when history is enabled.
recordings_dir: /dev/shm/xr-ai/recordings
gpu_id: 0

yaml/voice_gate.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# xr-render-demo voice gate configuration.
#
# magic_phrases — optional wake phrases during the active conversation.
#                 To require a "hey agent" / "agent" wake word, also set
#                 conversation.require_wake_phrase: true and use:
#                     magic_phrases:
#                       - "agent"
#                       - "hey agent"
# listening_chime — when true AND magic_phrases is non-empty, a short two-tone
#                   chime plays on fresh magic-phrase matches. Inert while
#                   magic_phrases is empty (no match can occur).
# followup_grace_s — after a bare wake phrase, the next utterance may begin
#                    within this window. Applies when wake gating is required.
mode: conversation
magic_phrases: []  # no wake word during the conversation
listening_chime: true
followup_grace_s: 5.0

# Start once, then speak naturally until the full stop phrase.
conversation:
  start_phrase: "Hey agent, let's start talking"
  stop_phrase: "Hey agent, let's stop talking"
  require_wake_phrase: false
  phrase_window_s: 6.0

# The default stack supplies speaker ASR; wait for it before worker readiness.
# For explicit batch STT, set speaker.enabled false and add an stt model entry.
speaker:
  enabled: true
  backend: required
  base_url: http://127.0.0.1:8102

yaml/xr_render_demo_worker.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# xr-render-demo worker configuration.
# Paths resolve relative to this file's directory.

# ── Voice gate ────────────────────────────────────────────────────────────────
# Interaction mode and optional wake phrases live in voice_gate.yaml.
# The sample starts with a closed conversation; the start phrase opens it.
voice_gate_yaml: voice_gate.yaml

# ── Native capability services ───────────────────────────────────────────────
scene_endpoint: tcp://127.0.0.1:8320
openxr_endpoint: tcp://127.0.0.1:8330
video_memory_endpoint: tcp://127.0.0.1:8310
# Past-moment questions ("what was I holding a moment ago?") need recorded
# video: enable video_recording in device_io_hub.yaml and point recordings_dir
# in video_memory_service.yaml at the same directory. Omitting this setting
# enables history. The sample sets it to false because hub recording is off;
# when history is off, vision_agent answers from the live camera only.
video_history_enabled: false
text_memory_dir: /dev/shm/xr-ai/text-memory

# ── Voice activity detection (Silero VAD, ONNX backend) ──────────────────────
# silence_duration: seconds of trailing silence that ends an utterance.
# min_speech: minimum cumulative speech before STT fires (filters short blips).
# silero_threshold: speech-probability gate (0..1). Lower = more sensitive
#   (catches quieter/faster speech). Default 0.5 often misses the onset of a
#   word; 0.3 captures it earlier.
silence_duration:  0.8
min_speech:        0.15
silero_threshold:  0.3

# ── Idle timeout ──────────────────────────────────────────────────────────
# Auto-cancel the voice pipeline after this many seconds with no user/bot
# speech. DISABLED by default (0 = off) — a quiet session stays connected
# indefinitely, which is what XR sessions usually want. Set a positive value
# (e.g. 300 for 5 min) to opt in; the worker then tears the pipeline down
# after that long idle. VoiceAgent owns the timeout.
idle_timeout_secs: 0

Model-server defaults#

clef_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# The pinned snapshot downloads into model_cache on first launch and is reused.
model_name: Cloudflare/clef-flash
model_revision: 17f0b0ad64efb65d273590632833508766b2aae6
model_cache: ../../models

host: 127.0.0.1
# Clef uses 8120, separate from the local model servers' 810x ports.
# Port inventory: docs/source/components/ai-services.md.
port: 8120
device: cuda:0
dtype: bfloat16
max_length: 4096
max_body_bytes: 1048576

embedding_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# embedding-server — vLLM configuration.
# OpenAI-compatible embeddings API at http://localhost:8109/v1  (served-model-name: "embed")
# Endpoint: POST /v1/embeddings  (input text → float vector)

model: nvidia/Nemotron-3-Embed-1B-BF16

host: "0.0.0.0"
port: 8109
# Optional. HF_TOKEN in the environment takes priority over this value.
hf_token: ""

# Shared weight cache — resolves relative to this YAML.
model_cache: ../../models

# Pin to a specific GPU (or comma-separated list for multi-GPU).
# Unset to use all GPUs visible to the process.
# cuda_visible_devices: "0"

# vLLM knobs.
max_num_seqs:           32
tensor_parallel_size:   1
max_model_len:          8192
# Pooling models do not allocate a generation KV cache, so their deployment
# budget remains a total-memory fraction rather than an explicit cache size.
# Low utilization by default — the 1B model is small and leaves headroom for
# the other vLLM servers (VLM, LLM) sharing the same GPU.  Raise to 0.30–0.40
# if this server runs on a dedicated GPU.
gpu_memory_utilization: 0.20
enforce_eager:          true
# Optional vLLM attention backend; omit for automatic selection.
# The tea sample's Spark profile selects TRITON_ATTN for GB10 compatibility.
# attention_backend: TRITON_ATTN

# vLLM runtime. If this key is omitted, the wrapper falls back to pip;
# the checked-in configuration below selects the NGC Docker container.
#   pip     — vllm CLI from this wrapper's venv (today's behavior)
#   docker  — `docker run nvcr.io/nvidia/vllm:<tag> vllm serve …`
# Both honor the same config keys above; only the runtime hosting vllm differs.
# docker mode requires Docker Engine + NVIDIA Container Toolkit and a
# `docker login nvcr.io` (or NGC_API_KEY in the environment).
vllm_backend: docker

# Container image used when vllm_backend: docker.  Override to pin a different
# tag, an internal mirror, or a custom build.
vllm_image: nvcr.io/nvidia/vllm:26.10.pre1-py3

magpie_tts_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# TTS server configuration — nvidia/magpie_tts_multilingual_357m.
# Auto-discovered by xr-ai-launcher and passed as --config to tts_server.
# OpenAI-compatible API at http://localhost:8104/v1

# NeMo TTS model name.
# nvidia/magpie_tts_multilingual_357m  — multilingual, NVIDIA Open Model License;
#                                         ~1 GB VRAM
model: nvidia/magpie_tts_multilingual_357m

# Pin to a specific HuggingFace commit so we always load the revision that was
# validated against the NeMo version in pyproject.toml.  The 2025-07 HEAD added
# a pt-BR tokenizer that requires newer NeMo API (locale_specific_punct kwarg).
# Remove this pin after upgrading nemo_toolkit to a version that supports the
# new model config.
model_revision: "34d7e40da85cabc97f92198889b65cea27bc7fd1"

# Inference device: "cuda", "cpu", or "auto" (CUDA if available, else CPU).
device: auto

port: 8104
host: "0.0.0.0"

# NeMo FastPitch/VITS native output sample rate.
sample_rate: 22050

# NeMo + HuggingFace weight cache — resolved relative to this YAML file.
model_cache: ../../models

nemotron_omni_llm_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Nemotron-3-Nano-Omni — vLLM configuration.
# Omni multimodal model supporting text + video input.
# OpenAI-compatible API at http://localhost:8108/v1  (served-model-name: "llm")
#
# GPU auto-selection:
#   Blackwell (SM100+) → model_blackwell  (NVFP4, --kv-cache-dtype fp8)
#   Ada, Hopper, Ampere → model_ada        (FP8,   --kv-cache-dtype fp8)
#   use_bf16: true     → model_bf16       (BF16,  no kv-cache quantisation)

model_blackwell: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4
model_ada:       nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8
model_bf16:      nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

# Set to true to force BF16 regardless of GPU (highest quality, most VRAM).
use_bf16: false

host: "0.0.0.0"
port: 8108
# Optional. HF_TOKEN in the environment takes priority over this value.
hf_token: ""

# Weight cache — resolves relative to this YAML.
model_cache: ../../models

# Pin to a specific GPU (or comma-separated list for multi-GPU).
# Unset to use all GPUs visible to the process.
# cuda_visible_devices: "0"

# vLLM knobs.
# The launcher defaults to 384 sequences, or 4 for the NVFP4 model when
# spark_uma is enabled. Set max_num_seqs to override that hardware-aware default.
tensor_parallel_size:   1
max_model_len:          32768
gpu_memory_utilization: 0.85
# Optional fixed KV-cache size in bytes per GPU. When this is set, vLLM still
# uses gpu_memory_utilization for its initial free-memory admission check.
# kv_cache_memory_bytes: 2147483648
enforce_eager:          false

# Video input — controls how video frames are sampled and pruned.
video_pruning_rate: 0.5   # fraction of redundant video tokens to drop
video_fps:          2     # FPS for video sampling
video_num_frames:   64    # max frames per video clip

# vLLM runtime. If this key is omitted, the wrapper falls back to pip;
# the checked-in configuration below selects the NGC Docker container.
#   pip     — vllm CLI from this wrapper's venv (today's behavior)
#   docker  — `docker run nvcr.io/nvidia/vllm:<tag> vllm serve …`
# Both honor the same config keys above; only the runtime hosting vllm differs.
# docker mode requires Docker Engine + NVIDIA Container Toolkit and a
# `docker login nvcr.io` (or NGC_API_KEY in the environment).
vllm_backend: docker

# Container image used when vllm_backend: docker.  Override to pin a different
# tag, an internal mirror, or a custom build.
vllm_image: nvcr.io/nvidia/vllm:26.10.pre1-py3

# DGX Spark is detected automatically to select the documented four-sequence
# limit and unified-memory cold-start safeguards. Set spark_uma to override.

nemotron3_nano_llm_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Nemotron-3-Nano LLM server configuration. The wrapper selects NVFP4 on
# Blackwell and FP8 on earlier supported GPU architectures.
# Auto-discovered by xr-ai-launcher (filename matches the
# 'nemotron3_nano_llm_server' process command) and passed as --config.
# Exposes an OpenAI-compatible API at http://localhost:8107/v1 via vLLM.
#
# Reasoning mode: the model's chat template defaults to
# ``enable_thinking=True``.  vLLM's ``--reasoning-parser nano_v3`` splits
# the resulting ``<think>…</think>`` into message.reasoning_content so the
# actual answer in message.content is never polluted — this is what keeps
# voice TTS from reading the reasoning aloud.

# The wrapper selects the checkpoint from the detected GPU compute capability.
model_blackwell: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
model_ada: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8

host: "0.0.0.0"
port: 8107
# Optional. HF_TOKEN in the environment takes priority over this value.
hf_token: ""

# Shared weight + reasoning-parser cache.  Resolves relative to this YAML;
# ../../models lands in the repo-root models/ directory shared by the other
# model services.
model_cache: ../../models

# Pin to a specific GPU (or comma-separated list for multi-GPU).
# Also controls which GPU the compute-capability query targets for model selection.
# Unset to use all GPUs visible to the process.
# cuda_visible_devices: "0"

# vLLM server knobs.
#
# On Blackwell (B200 / RTX PRO 6000 96 GB): NVFP4 weights are ~20 GB. The
# checked-in 0.85 value gives a dedicated server room for its KV cache; lower
# it in a reviewed GPU profile when sharing the device with another model.
#
# On Ada Lovelace (RTX 6000 Ada 48 GB): FP8 weights ~30 GB; run the LLM
# on a dedicated GPU and keep the vlm-server + stt-server on the second GPU.
# With LLM alone on a 48 GB card, 0.85 (~41 GB) gives ~11 GB for KV cache.
# The launcher defaults to 8 sequences, or 4 when spark_uma is enabled for
# DGX Spark. Set max_num_seqs to override that hardware-aware default.
tensor_parallel_size: 1
max_model_len:        32768

gpu_memory_utilization: 0.85

# Skip CUDA graph capture + FlashInfer MoE autotune at startup.  For
# NemotronH-30B-A3B-NVFP4 those two phases are silent (no log output)
# and take 3-8 min on first run — bad UX for a voice agent where the
# user is waiting for the server to become healthy.  Eager mode starts
# in ~5 s after weight load and is 10-20% slower per token, which is
# imperceptible for this workload (<250 tokens per turn, STT+VAD+TTS
# already dominate turn latency).  Set to ``false`` if you want
# maximum throughput and can wait for graph capture to finish.
enforce_eager: true

# vLLM runtime. If this key is omitted, the wrapper falls back to pip;
# the checked-in configuration below selects the NGC Docker container.
#   pip     — vllm CLI from this wrapper's venv (today's behavior)
#   docker  — `docker run nvcr.io/nvidia/vllm:<tag> vllm serve …`
# Both honor the same config keys above; only the runtime hosting vllm differs.
# docker mode requires Docker Engine + NVIDIA Container Toolkit and a
# `docker login nvcr.io` (or NGC_API_KEY in the environment).
vllm_backend: docker

# Container image used when vllm_backend: docker.  Override to pin a different
# tag, an internal mirror, or a custom build.
vllm_image: nvcr.io/nvidia/vllm:26.10.pre1-py3

# DGX Spark is detected automatically to cap max_num_seqs at 4 and enable the
# unified-memory cold-start safeguards. Set spark_uma explicitly to override.

pocket_tts_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Pocket TTS server reference configuration.
# Copy to your agent sample root as pocket_tts_server.yaml and edit as needed.
#
# This release accepts only bill_boerst. Its source Voice-Zero recording is CC0.

voice:       bill_boerst
language:    english
# cuda, cpu, or auto (cuda when available, otherwise cpu).
device:      cuda
port:        8105
startup_timeout_s: 600
model_cache: ../../models

speaker_stt.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Loopback HTTP serves health/model identity; a private WebSocket carries PCM.
host: 127.0.0.1
port: 8102
asr_model: nvidia/multitalker-parakeet-streaming-0.6b-v1
diar_model: nvidia/Nemotron-3-Diarization
device: cuda
precision: bf16
# Hardware profiles or CUDA_VISIBLE_DEVICES own GPU placement.
model_cache: ../../models
# Each session holds one diarization cache and up to eight candidate ASR caches.
# After enrollment, diagnostics add temporary ASR caches for non-owner speakers.
max_sessions: 16

stt_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# STT server configuration — nvidia/parakeet-tdt-0.6b-v3.
# Auto-discovered by xr-ai-launcher and passed as --config to stt_server.
# OpenAI-compatible API at http://localhost:8103/v1

# NeMo model name.
# nvidia/parakeet-tdt-0.6b-v3  — 0.6B TDT model; CC-BY-4.0; ~1.5 GB VRAM
# nvidia/parakeet-rnnt-1.1b    — stronger, larger; same CC-BY-4.0 license
model: nvidia/parakeet-tdt-0.6b-v3

# Inference device: "cuda", "cpu", or "auto" (CUDA if available, else CPU).
device: auto

port: 8103
host: "0.0.0.0"

# A cold start imports NeMo, downloads weights when absent, initializes CUDA,
# and loads the model before /health opens. Use a positive finite number, and
# increase it on slow links.
startup_timeout_s: 600

# NeMo + HuggingFace weight cache — resolved relative to this YAML file.
model_cache: ../../models

vlm_server.yaml#

# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# vlm-server — vLLM configuration for the Cosmos3 Nano Reasoner.
# OpenAI-compatible API at http://localhost:8100/v1  (served-model-name: "vlm")
# Accepts image_url content blocks (base64 data URLs or HTTP URLs).
#
# The architecture override selects the Reasoner runtime; rationale and loader
# details: docs/source/components/ai-services.md#per-server-notes.

model: nvidia/Cosmos3-Nano
hf_overrides:
  architectures:
    - Cosmos3ForConditionalGeneration

host: "0.0.0.0"
port: 8100
# Optional. HF_TOKEN in the environment takes priority over this value. Only
# needed for gated models or to lift HF download rate limits. See
# docs/source/getting_started/credentials.md.
hf_token: ""

# Shared weight cache — resolves relative to this YAML.
model_cache: ../../models

# Pin to a specific GPU (or comma-separated list for multi-GPU).
# Unset to use all GPUs visible to the process.
# cuda_visible_devices: "0"

# vLLM knobs — VLMs are more memory-intensive than text-only LLMs.
max_num_seqs:           4
tensor_parallel_size:   1
max_model_len:          8192
gpu_memory_utilization: 0.85
# Optional fixed KV-cache size in bytes per GPU. When this is set, vLLM still
# uses gpu_memory_utilization for its initial free-memory admission check.
# kv_cache_memory_bytes: 1610612736
enforce_eager:          true
# Recommended by NVIDIA for the Cosmos3 Reasoner vLLM path.
async_scheduling:       true
mm_encoder_tp_mode:     data

# Optional attention overrides. When omitted, Cosmos3 on SM12 GPUs (RTX
# Blackwell and GB10) with the default Docker image uses TRITON_ATTN for both
# decoder and vision attention to reduce cold-start latency. Other GPUs,
# custom images, and pip mode retain vLLM defaults.
# Set either to auto to opt back into vLLM's selection for that component.
# attention_backend: TRITON_ATTN
# mm_encoder_attn_backend: TRITON_ATTN

# Max images accepted per request.
# Raise max_model_len as needed for larger multi-image prompts.
max_images_per_prompt:  4

# Max video items accepted per request. Default 0 (disabled).
# Multimodal models reserve activation memory at startup for the maximum video
# feature size. Leave at 0 unless your worker sends video_url content blocks.
max_videos_per_prompt:  0

# vLLM runtime. If this key is omitted, the wrapper falls back to pip;
# the checked-in configuration below selects the NGC Docker container.
#   pip     — vllm CLI from this wrapper's venv (today's behavior)
#   docker  — `docker run nvcr.io/nvidia/vllm:<tag> vllm serve …`
# Both honor the same config keys above; only the runtime hosting vllm differs.
# docker mode requires Docker Engine + NVIDIA Container Toolkit. NVIDIA's
# public vLLM image supports anonymous pulls; NGC login is only needed for a
# restricted override image (and can be supplied through NGC_API_KEY).
vllm_backend: docker

# Container image used when vllm_backend: docker.  Override to pin a different
# tag, an internal mirror, or a custom build.
vllm_image: nvcr.io/nvidia/vllm:26.10.pre1-py3