XR AI#

Build AI agents that see and hear what your users experience in XR, and respond in real time.

Tip

Try xr-ai with a coding agent. Paste this to run the browser demo:

Run the xr-ai Simple VLM browser demo for me. Reuse the checkout and ref
selected in this conversation or the current xr-ai working directory;
otherwise ask whether to use the latest release
(newest stable tag; a prerelease only if no stable exists) or main. Fetch
skills/run-demo/SKILL.md at that ref from
https://raw.githubusercontent.com/NVIDIA/xr-ai/. If it is absent at a
release tag, get my OK before using main for the skill, checkout, and docs.
Install the skill (or just follow it), handle any setup it requires, and
leave the demo running. Return the browser link, interaction verification
status, and stop instructions.

For development setup and other workflows, refer to Skills.

XR AI is an open-source stack that connects web, iOS/visionOS, AR-glasses, and XR-headset clients to GPU-accelerated AI services and tool-using agents. An agent can perceive live physical context, call native tools, and reply with audio or data in the same session. For remote-rendered AR and XR, XR AI integrates NVIDIA CloudXR, as the xr-render-demo sample shows.

It is especially useful when you need to:

  • Build multimodal XR agents that see, hear, reason, use tools, and respond in real time.

  • Target multiple client platforms — web, iOS/visionOS, AR glasses, and XR headsets.

  • Use NVIDIA open models out of the box while keeping the freedom to bring your own.

  • Deploy wherever NVIDIA GPUs run — cloud, data center, workstation, or edge.

  • Integrate NVIDIA CloudXR for remote-rendered AR and XR, as the xr-render-demo sample shows.

  • Keep transport, rendering, model services, tools, and agent logic separated so each layer evolves independently.

🤖 Try xr-ai with your agent

Launch the browser demo with run-demo and receive its connection link and verification status.

Skills
🚀 Get started

Try the recommended conversation sample, explore the legacy Simple VLM browser demo, or launch the full xr-render demo.

Quickstart
🧩 Architecture

How clients, DeviceIOHub, agents, models, services, and the launcher fit together.

Architecture
🛠️ Components

The server runtime, agent SDK, AI services, and the launcher.

Components
📦 Build your application

Start from the recommended conversation-example and choose the SDK layers your application needs.

Build your application
🧱 Add a sample

Prepare an application for contribution to the repository’s sample collection.

Adding a new sample