Architecture#

Overview#

As your application adds perception, conversation, actions, or XR rendering, you need to decide where each capability belongs and how it connects to the others. XR AI separates these responsibilities so you can extend your application’s behavior while reusing its device connections and model services.

System context#

The device I/O hub, workers, model servers, and optional capability services run as independent processes and communicate through explicit interfaces.

                                     control plane
                          sample orchestrator + launcher
                         starts, orders, and monitors processes
                                         |
                                         v
+---------------+   media and data   +----------------+   IPC events   +----------------+
| Clients       | <----------------> | DeviceIOHub    | <------------> | agent workers  |
| (web/mobile/XR)|                   | + transport    |                | + agent SDK     |
+---------------+                    +-------+--------+                +-------+--------+
                                                |
                                     normalized | media and return data
                                                v
                                       +-----------------+
                                       | optional capture |
                                       +-----------------+
                                                                            |
                                               typed model and tool calls   |
                                                +---------------------------+--------+
                                                |                                    |
                                                v                                    v
                                      +-------------------+               +--------------------+
                                      | AI model services |               | capability services |
                                      | local or hosted   |               | or local tools     |
                                      +-------------------+               +--------------------+

The architecture has four cooperating planes:

Media plane: DeviceIOHub receives client audio, video, and data through a transport connector, then fans participant-tagged events out to workers and optional passive processors such as session capture.
Application plane: agent workers own application behavior, participant state, model and tool orchestration, concurrency, and cancellation.
Service plane: typed model clients and tools isolate workers from model hosting and capability-process protocols.
Control plane: each sample’s orchestrator uses xr-ai-launcher to start processes in dependency order and monitor their readiness and lifetime.

These planes are boundaries of responsibility, not required deployment hosts. They can run on one machine, while model endpoints can also be persistent local services or externally hosted APIs.

Components and ownership#

Component	Owns	Does not own
XR clients	Device capture, presentation, and user interaction	Agent execution or service orchestration
Transport connector	Transport-specific sessions and conversion to hub events	Agent-facing APIs or application policy
DeviceIOHub	Media fan-out, participant identity, return routing, and shared-media access	Agent state, model calls, or tools
Session capture	Timestamped raw media/events and pluggable demo or workflow projections	Transport sessions, agent policy, model inference, or live-path flow control
Agent SDK	Lightweight hub IPC, typed runtime events, model protocols, voice composition, and tool primitives	Application lifecycle and decision policy
Agent worker	Application state, tasks, prompts, model and tool loops, concurrency, and cleanup	Transport internals or model-server lifecycle
AI model services	Inference and model-specific serving behavior	Participant routing or application policy
Tools and capability services	Bounded application capabilities	General agent orchestration
Sample orchestrator	Process declarations, startup ordering, readiness, and shutdown	Runtime business logic

The separation lets a worker change model deployment or client transport without rewriting its application logic. It also keeps heavy transport and model dependencies out of the minimal agent-to-hub IPC package.

Runtime data paths#

Client media and participant events#

A client joins through the configured transport and publishes audio, video, and data. It may instead keep video off and expose an opt-in still-image capture handler.
The transport connector converts that input into participant-tagged hub events. Audio, data, and participant events travel inline; video pixels stay in shared memory and frame notifications remain lightweight.
DeviceIOHub fans subscribed events out to ProcessorEndpoint consumers. Multiple agents or passive processors can subscribe to the same input.
A worker requests video pixels only when its application needs a frame. If no fresh video exists, a participant-bound tool can request one encoded still from the originating client; the client returns it in a targeted, correlated byte stream.
Return audio and data name the originating participant. The hub validates the target and the connector delivers the response only to that participant.

An optional capture process can observe both sides after hub normalization. It uses the same on-demand video path as other passive processors and never joins the transport session, so encoder load remains outside the hub’s latency-sensitive path. Its shared session writer can follow participant lifetime or accept private start/stop commands from an agent adapter; demo and raw workflow projections reuse the same timestamped media and metadata core.

The resulting portable contract is:

participant -> hub -> subscribed worker -> hub -> same participant

LiveKit currently provides the client transport, but it remains behind the hub boundary. Workers communicate only through XR AI’s msgpack over ZMQ IPC and do not import or address LiveKit directly. Refer to Server runtime for shared-memory behavior, topics, and transport implementation details.

Agent, model, and tool execution#

A voice-capable application commonly composes this path:

audio -> voice input -> STT -> application agent -> LLM/VLM -> voice output -> TTS
                                  |                 ^                         |
video -- on-demand frame access --+                 |                         |
                                  +---- tools ------+                         v
                                                               return audio and data

This is composition rather than a fixed pipeline. An application may consume text instead of speech, omit models, execute deterministic tools locally, call typed capability services, or publish events for another in-process agent registered with the same AgentRuntime to consume.

Voice interaction policy is separate from ASR selection. always_on accepts all recognized speech, wake_only accepts addressed turns, conversation requires a start phrase before natural dialogue, and hybrid also accepts closed-state addressed one-off questions without enrolling an owner. With speaker ASR, the service attributes each candidate before authorizing a turn; only an unambiguous start phrase establishes the persistent conversation owner. Batch ASR applies the same interaction policy without individual speaker identity. Refer to Conversation controls.

Workers obtain LLM, VLM, STT, TTS, and embedding clients from xr_ai_models. Model profiles keep three concerns separate:

adapter behavior and model-specific wire details;
endpoint location, credentials, and health behavior;
optional deployment metadata. Consumer profiles omit it and leave endpoint lifecycle to the operator. Sample-local model-server convenience launchers declare their server selection separately and share runtime defaults under utils/xr-ai-model-server-stack/.

Tools are ordinary in-process Tool or AsyncTool objects. A tool can perform local work or call a typed service, but application agents retain ownership of tool selection, task lifetime, retries, and participant context. Refer to Agent SDK and AI services for the concrete interfaces and profiles.

Process and deployment model#

Each sample is an executable process graph declared by a small orchestrator. DeviceIOHub always runs as its own process; workers and capability services run separately so their dependencies, failures, and cleanup remain isolated.

Stack items start in declaration order; members of a Parallel item start concurrently, and the next item waits for every member to signal ready. Each ready-file reports only that process’s initialization. A premature process exit fails the stack and triggers coordinated shutdown. Ready-files order process startup; they do not determine whether a client may connect.

Each sample’s model-servers/ convenience launcher selects the local inference services it needs and keeps their weights loaded across agent restarts. Shared server defaults are combined with sample-owned hardware overrides. Startup waits for readiness; stop targets only that sample’s configured service names and ports using the existing ownership guards. Application launchers separately declare their hub, workers, and capability services.

Worker model profiles configure client adapters and endpoints, not the local model launch plan. Speaker-ASR selection and its endpoint live in the worker’s voice-gate YAML. Independently deployed endpoints, including native NIM, remain consumer configuration choices; skip the local model launcher when all model services are external. For model availability and worker startup behavior, refer to Consumer model readiness. Detailed local startup, shutdown, and persistence belongs in Launcher and process model.

Architectural invariants#

The following constraints define the supported system boundary:

One hub, many clients, many agents. The hub may fan one participant’s input out to several consumers, but it never uses the return path to route one participant’s data to another.
Transport details stop at the hub. Workers use xr_ai_hub; transport SDKs and server packages do not enter agent APIs.
Capture is transport-independent and policy-separated. Session capture consumes normalized hub IPC and routed return traffic in a separate process. Participant-lifetime or agent-controlled boundaries select what the shared writer retains; demo rendering and raw workflow packets are projections over that writer. Agent wrappers own the capture namespace and fixed metadata; model-facing start and stop actions do not accept filesystem paths. Bounded capture queues may drop recording frames but must not apply backpressure to the hub, clients, or agents.
Raw media stays on the media path. Video pixels remain in shared memory until explicitly requested, and raw media is not embedded in runtime events or tool results.
Applications own behavior and resources. An agent owns its state, background tasks, lifecycle, queues, cancellation, and concurrency policy. Shared runtime packages provide delivery and composition rather than taking over that ownership.
Models are reached through typed factories. Vendor protocols and model-specific quirks remain behind xr_ai_models, rather than spreading through workers.
Process dependencies are explicit. Orchestrators declare startup order, readiness, and ownership instead of relying on import-time side effects or an implicit global runtime.
Process and client readiness are distinct. Ready-files order launcher items and report only process readiness. DeviceIOHub owns client readiness. Agents opt in with announces_readiness=True, report only their own _agent.status, and gate only subscribed participants; the hub aggregates their status per participant. Passive processors do not gate clients.

These constraints are the stable architecture. Individual transports, model backends, tools, and sample workflows are replaceable implementations within it.

Extension points#

To add or replace	Extend at this boundary
XR client	Join through the client transport and consume participant-scoped return media and data
Transport	Implement the connector side of the hub IPC contract
Agent application	Add a worker that uses `xr_ai_hub` and the relevant SDK packages
Model or provider	Add model configuration and an adapter behind the typed model protocols
Local capability	Implement an in-process tool
Shared or isolated capability	Implement a typed service and a service-backed tool
Managed process	Add a launcher `Process` entry and signal readiness from the process

For the repository conventions and concrete file layout used by new samples, refer to Adding a sample.

Where details live#

This architecture overview intentionally stops at system structure and contracts. Use these references for implementation and operational detail:

Topic	Authoritative page
Hub IPC, shared memory, participant isolation, and LiveKit integration	Server runtime
SDK package boundaries, agents, voice, runtime events, and tools	Agent SDK
Model protocols, deployment profiles, NIM, vLLM, and persistent servers	AI services
Process ordering, readiness, launch modes, and shutdown	Launcher and process model
External ports, TLS, proxies, and firewall requirements	Networking and firewall
API keys and credential storage	Credentials