Build your application#

Overview#

Build your own XR AI application from a sample and keep its code and configuration under apps/. This gives you an application workspace where you can develop your task-specific behavior while using the checkout’s SDK and services.

Start from a source checkout#

The directions below assume the application is under apps/<your-app>/. The XR AI SDK distributions are not published to a package registry; they resolve through relative [tool.uv.sources] entries in each application’s pyproject.toml.

Use the latest stable release by default, falling back to the latest prerelease when no stable release exists. Use main when the application needs unreleased changes. The documentation and checkout must use the same ref: use https://nvidia.github.io/xr-ai/<ref>/, replacing <ref> with the selected release tag or main. The /latest/ alias can fall back to main when a release lacks current entry points, so it does not guarantee a match with a release checkout.

Create the checkout:

git clone https://github.com/NVIDIA/xr-ai.git
cd xr-ai

To use the latest stable release, or the latest prerelease when no stable release exists, run:

release_tag="$(git tag --list 'v*' | python3 .github/scripts/select_latest_docs_release.py)"
if [ -n "$release_tag" ]; then
  git checkout "$release_tag"
else
  echo "No release tag found; continuing on main." >&2
fi

After checking out a release, confirm that docs/source/guides/building-your-app.md and skills/build-your-app/SKILL.md exist in the checkout. If either is absent, ask before switching the documentation, skill, and checkout to main.

Read AGENTS.md, then review the system requirements and credential sources before resolving the application. Choose local model services or hosted endpoints. If setup already selected one, keep that choice. The copied reference application uses local model services by default. For hosted inference, refer to Deploying samples with NIM endpoints; the hub still needs NVENC and NVDEC on its host as described in the system requirements.

Choose the layers your application needs#

Package	Use it for
xr-ai-launcher Import: `xr_ai_launcher` Source: `utils/xr-ai-launcher`	Process management for starting DeviceIOHub, workers, and application services in dependency order, with only `xr-ai-gpu` as a runtime dependency.
xr-ai-hub-client Import: `xr_ai_hub` Source: `agent-sdk/xr-ai-hub`	IPC with DeviceIOHub, participant events, data and audio messages, and live frame access.
xr-ai-logging Import: `xr_ai_logging` Source: `utils/xr-ai-logging`	Shared logging setup used by the copied orchestrator and worker.
xr-ai-models Import: `xr_ai_models` Source: `agent-sdk/xr-ai-models`	Typed LLM, VLM, STT, TTS, and embedding clients selected from deployment profiles.
xr-ai-agent-runtime Import: `xr_ai_runtime` Source: `agent-sdk/xr-ai-runtime`	Typed agent lifecycle, registration, publication, and participant-scoped subscriptions.
xr-ai-tools Import: `xr_ai_tools` Source: `agent-sdk/xr-ai-tools`	Typed `Tool` and `ToolSet` objects. Add `relay` for `run_tool_loop()`, `frames` for live-frame selection, and `vision` for VLM query tools.
xr-ai-voice Import: `xr_ai_voice` Source: `agent-sdk/xr-ai-voice`	The `VoiceAgent` runtime for hub transport, STT, voice gating, TTS, interruption, readiness, and cleanup.
xr-ai-voicegate Import: `xr_ai_voicegate` Source: `utils/xr-ai-voicegate`	Wake-phrase and follow-up-turn gating used by the copied voice application.
xr-ai-web-events Import: `xr_ai_web_events` Source: `agent-sdk/xr-ai-web-events`	A bounded browser view for compact application events explicitly selected by the application.

The distribution names, import names, and source directories are not always identical. In particular, xr-ai-hub-client comes from agent-sdk/xr-ai-hub, and xr-ai-agent-runtime comes from agent-sdk/xr-ai-runtime. Refer to Agent SDK for package ownership boundaries and composition guidance.

Copy the Conversation application#

agent-samples/conversation-example/ combines conversation, visual questions, one-shot alerts, and note-taking. Copy its tracked files from the repository root and rename its Python package and worker configuration. Keep only the destinations your application needs. The deprecated Simple VLM sample remains available for the browser demo and existing integrations.

mkdir -p apps/my-app
git archive HEAD agent-samples/conversation-example |
  tar -x --strip-components=2 -C apps/my-app
mv apps/my-app/worker/conversation_example_worker apps/my-app/worker/my_app_worker
mv apps/my-app/yaml/conversation_example_worker.yaml apps/my-app/yaml/my_app_worker.yaml

Replace the sample names consistently:

Existing name	Application name
`conversation_example_worker`	`my_app_worker`
`conversation-example`	`my-app`
`conversation_example`	`my_app`

Substitute the application’s real kebab-case name for my-app, then derive its snake-case package and command names mechanically. Apply the replacements in the listed order across the entire copied tree, including comments and docstrings. Update any remaining agent-samples/ working-directory references to apps/, and change prose that still describes the application as a repository sample. The following check should produce no output:

rg -ni 'conversation[-_ ]example' apps/my-app

The apps/ and agent-samples/ directories are at the same depth, so the copied projects’ relative sources continue to work. Keep each [tool.uv.sources] entry paired with its distribution in [project].dependencies. Every unpublished XR AI dependency must retain a repository source mapping.

Use this map to separate reusable scaffolding from the sample’s capabilities:

Files	Reuse	Replace or adapt
`main.py` and the root `pyproject.toml`	Keep the `run_stack` entry point, DeviceIOHub process, worker process, and launcher source entries.	Rename the project, command, logging namespace, worker command, and worker YAML path. Update `project.description` to describe the application. Add only application-owned service processes.
`yaml/device_io_hub.yaml`	Keep the typed DeviceIOHub configuration shape.	Set the room, ports, web client, and network behavior. Refer to the DeviceIOHub reference YAML for coupled port and URL changes.
`worker/pyproject.toml`, `__init__.py`, and `__main__.py`	Keep the named-package layout, console entry point, argument parsing, and delegation to `run_app()`.	Rename the distribution, package, and entry point. Remove dependencies only after their imports and features are gone.
`worker/.../config.py` and `yaml/*_worker.yaml`	Keep typed loading and paths resolved relative to the YAML file.	Replace the prompt, frame, model, voice, and application-specific settings with the fields the application consumes.
`worker/.../app.py`	Keep the `AgentRuntime`, `AgentRouter`, and `VoiceAgent` composition and lifecycle for the capabilities retained.	Adapt registered destinations, model roles, health probes, output topics, and context subscriptions together. Keep the router decision service while using routing.
`worker/.../_visual_agents.py`, `_alert_agent.py`, `_alert_checks.py`, `_alert_compile.py`, `_note_taking.py`, and `prompts/`	Keep participant-scoped ownership and cancellation patterns. Refer to Conversation example for their boundaries.	Adapt or remove the current-view, alert, and note-taking destinations and their prompts, tools, and subscriptions. Do not leave removed destinations registered.
`yaml/models.json`	Keep the adapter and endpoint separation.	Declare only the logical model roles, client adapters, and shared endpoints the application needs. Do not add model-server deployment or readiness settings.
`yaml/voice_gate.yaml`	Keep it when the application uses voice.	Choose conversation controls or direct speech using Conversation controls, and test the application’s first spoken request.
`README.md`	Keep a short configure and run path.	Describe the application, its owned and reused processes, settings, and exact start commands.

Rewrite the copied README as application-owned documentation. Replace its title and the “Conversation example” and “This sample” descriptions, and remove links to the Conversation sample reference. In particular, make sure the name replacements did not leave a nonexistent /reference/my-app.html link. Change its working directory to apps/my-app/ and use the model-server command shown below. The copied model-servers/ directory is an application-owned preset if you choose to maintain it; the commands below reuse the repository’s Conversation stack without changing its configuration.

The copied DeviceIOHub YAML leaves credentials blank, so the hub generates a fresh pair at startup. To use a fixed pair, set LIVEKIT_API_KEY and LIVEKIT_API_SECRET in the launch environment. Never put actual keys or tokens in source files. Refer to Credentials for the supported credential sources.

Frame selection and vision inference use separate public paths: import live-frame selection from xr_ai_tools.current_frame and image inference from xr_ai_tools.vision.

Resolve and run the application#

From apps/my-app/, resolve both projects before starting the stack:

uv --config-file ../../uv.toml sync
uv --config-file ../../uv.toml sync --project worker

Local model services#

The model launcher requires HF_TOKEN. Configure it through one of the sources in Credentials before starting the application. The copied application reuses speaker ASR, LLM, VLM, TTS, and the Clef decision service. Start those shared services first and wait for the launcher to report that all processes are ready:

uv --config-file ../../uv.toml run \
  --project ../../agent-samples/conversation-example/model-servers conversation_example_model_servers

Hosted endpoints#

Follow Deploying samples with NIM endpoints to configure and validate the STT, LLM, VLM, and TTS roles in the application’s yaml/models.json. Retain the router role and start Clef separately or use a compatible decision endpoint as that guide describes. Keep a copy of the local profile before replacing its entries. Apply the guide’s voice-gate settings for hosted speech to disable the copied local speaker backend. Start the application after endpoint validation.

Start the application#

After the selected model environment is ready, start the application from apps/my-app/:

uv --config-file ../../uv.toml run my_app

Open the full web-client URL printed by DeviceIOHub, including its scheme and port. For a client on another device, replace localhost with the XR AI host’s IP address or hostname reachable by that device. HTTPS is enabled by default. Refer to TLS for clients for certificate trust on the client device and Browser reports an empty response on the hub port if the browser reports an empty response.

For the copied voice-and-vision application, select Connect, grant camera and microphone access, choose On-demand images or Live video under Camera Mode, then select Start Microphone. Use the conversation controls configured in yaml/voice_gate.yaml before speaking the application’s request. The web client’s Send control delivers a typed query to the agent and bypasses speech gating. It checks the query path without exercising STT or conversation controls.

Write the application’s README with its intended client, exact connection steps, voice mode, and an example request. Check the adapted commands and link targets before handing the application to its user.

The application launcher owns only DeviceIOHub, the worker, and any other application processes. The separate model-server launcher owns shared model startup and readiness. The voice runtime checks required speaker ASR readiness, and the copied worker checks LLM, router, and VLM health before announcing readiness; it does not perform VLM inference as a startup warmup. Adapt those probes to the capabilities the application retains. When a model role changes, update yaml/models.json and the worker’s service construction. Do not add shared model services to the application’s PROCESSES list. If a process fails, use its launch context and run-log directory to investigate through Troubleshooting.

Verify without hardware#

Turn each requirement in the application brief into an observable acceptance case. Include configuration defaults when a key is omitted, the intended first request, and the recovery behavior for unavailable inputs. Keep these checks in apps/my-app/tests/ using the layout and command below.

From the repository root, load the worker configuration before starting services. This check imports the renamed package, reads the application YAML, and resolves packaged prompt files:

uv --config-file uv.toml run --project apps/my-app/worker \
  python - <<'PY'
from pathlib import Path
from my_app_worker.config import load_config

load_config(Path("apps/my-app/yaml/my_app_worker.yaml"))
PY

Exercise model wire behavior with tests/_stub_openai.py, which provides an httpx.MockTransport implementation of OpenAI-compatible endpoints. The wire tests exercise xr_ai_models clients without importing the application worker. The deprecated sample’s tests/test_simple_vlm_example_wire.py remains a small STT, VLM, and TTS wire-format test pattern. Use tests/test_xr_render_demo_wire.py for LLM tool-call flows. For the copied application’s routing, context, and destination behavior, adapt the CPU cases in tests/test_conversation_example_worker.py and tests/test_conversation_example_agents.py; use a fake decision service rather than starting Clef for these checks.

The hub, make_connector, and make_processor fixtures in tests/conftest.py run DeviceIOHub IPC over local ZMQ sockets. They demonstrate participant routing and worker-facing endpoint coverage without LiveKit, a camera, a microphone, Docker, or a GPU. Pytest does not load that file for tests under apps/my-app/ because it is not an ancestor of the application test directory. Keep application-owned fixtures in apps/my-app/tests/conftest.py; copy and adapt only the repository fixtures the application needs.

Add the application’s pytest settings to apps/my-app/pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"
pythonpath = ["worker"]
markers = [
  "gpu: requires local GPU, Docker, or NVENC",
  "integration: starts a real service process",
]

Point model clients at a StubOpenAI-style transport, then run the application test directory explicitly with the repository test environment:

uv --config-file uv.toml run --project tests \
  pytest -v apps/my-app/tests -m "not gpu"

Refer to Testing for the complete test commands and marker rules.

Before reporting that the application is complete, match each requirement to its test or client-interaction result. Exercise the intended first request through a real client after the hardware-free checks pass. Record the selected model environment, configuration, observed response, and any simulated media or unverified requirement. Distinguish typed input from spoken input and record whether STT and the configured conversation controls were exercised. Passing tests and service readiness do not replace that client check.

Understand the application boundary#

Repository file checks exclude the top-level apps/ directory. Application projects do not enter DEPENDENCIES.md or the dependency manifest, and the repository’s Ruff and SPDX checks do not inspect application-owned files. The sample documentation, configuration catalogs, and test discovery remain scoped to agent-samples/ and tests/. Git ignores apps/* by default so private application work is not staged accidentally. Remove that ignore rule when the application owner wants to track the application in the fork.

The rest of the checkout remains repository-owned. Commit-wide hooks such as DCO sign-off still run when installed, and changes outside apps/ continue to follow AGENTS.md and the repository’s contribution checks.