XR AI#

Build AI agents that see and hear what your users experience in XR, and respond in real time.

Tip

Using a coding agent? Paste this to it:

Set up xr-ai for me: ask me whether to build against the latest release
(newest stable tag; a prerelease only if no stable exists) or main, then
fetch skills/getting-started/SKILL.md at that ref from
https://raw.githubusercontent.com/NVIDIA/xr-ai/. If the file does not exist
at that ref, the release predates agent setup: get my OK to use main for
both the skill and the checkout. Install the skill (or just follow it), and
walk me through the rest of the setup.

XR AI is an open-source stack that connects web, iOS/visionOS, AR-glasses, and XR-headset clients to GPU-accelerated AI services and tool-using agents. An agent can perceive live physical context, call native tools, and reply with audio or data in the same session. For remote-rendered AR and XR, XR AI integrates NVIDIA CloudXR, as the xr-render-demo sample shows.

It is especially useful when you need to:

  • Build multimodal XR agents that see, hear, reason, use tools, and respond in real time.

  • Target multiple client platforms — web, iOS/visionOS, AR glasses, and XR headsets.

  • Use NVIDIA open models out of the box while keeping the freedom to bring your own.

  • Deploy wherever NVIDIA GPUs run — cloud, data center, workstation, or edge.

  • Integrate NVIDIA CloudXR for remote-rendered AR and XR, as the xr-render-demo sample shows.

  • Keep transport, rendering, model services, tools, and agent logic separated so each layer evolves independently.

🤖 Set up with your agent

The primary path: one pasted prompt sets a coding agent up with the xr-ai skills, and it does the rest.

Skills
🚀 Get started

Run a sample in minutes: the model servers, the simple VLM agent, or the full xr-render demo.

Quickstart
🧩 Architecture

How clients, DeviceIOHub, agents, models, services, and the launcher fit together.

Architecture
🛠️ Components

The server runtime, agent SDK, AI services, and the launcher.

Components
📦 Build your application

Start from the reference application and choose the SDK layers your app needs.

Build your application
🧱 Add a sample

Prepare an application for contribution to the repository’s sample collection.

Adding a new sample