Agent SDK#

Overview#

The Agent SDK helps you build applications that respond to what users see and say. It provides reusable support for device input, model calls, tools, conversation, and voice output, so you can focus on the behavior specific to your application.

Packages#

Packages separate dependency and ownership boundaries. The table maps source directories to Python imports; use distribution names for installation.

Directory	Import	Responsibility
`xr-ai-hub`	`xr_ai_hub`	Minimal IPC with DeviceIOHub using msgpack over ZMQ
`xr-ai-interaction`	`xr_ai_interaction`	Transport-neutral query, response, presence, and interruption messages
`xr-ai-models`	`xr_ai_models`	Typed model protocols, profiles, and OpenAI-compatible clients
`xr-ai-routing`	`xr_ai_routing`	Participant-scoped routing and the `RoutableAgent` contract
`xr-ai-conversation`	`xr_ai_conversation`	Routable conversation agent with direct response output, named context, and rolling-summary compaction
`xr-ai-runtime`	`xr_ai_runtime`	Agent registration and typed participant-scoped fan-out
`xr-ai-tools`	`xr_ai_tools`	Relay-managed tools and model tool-call helpers
`xr-ai-voice`	`xr_ai_voice`	Voice agent, transport, and private media pipeline
`xr-ai-web-events`	`xr_ai_web_events`	Bounded live browser views over selected application events

Refer to these versioned package guides: xr-ai-hub, xr-ai-interaction, xr-ai-models, xr-ai-routing, xr-ai-conversation, xr-ai-runtime, xr-ai-tools, xr-ai-voice, and xr-ai-web-events. Refer to the generated API Reference for public names, signatures, types, defaults, fields, and method-level behavior.

Ownership boundaries#

An Agent owns its state, resources, background tasks, lifecycle, and concurrency policy. It exposes ordinary Tool or AsyncTool instances for direct execution and model tool-call loops. Bounded turns use ToolSet and run_tool_loop(); callers retain model configuration, conversation policy, retries, participant context, cancellation, and task ownership. AgentRuntime owns only typed publication and delivery tasks; it does not own model loops, planning, memory, media, or agent-created work. A publication waits for its subscribers, so a subscriber that performs lengthy work must hand the work to its own bounded queue.

Workers construct model clients with the factories in xr_ai_models and depend on the corresponding typed service protocols. Adapter behavior, endpoint connectivity, deployment ownership, credentials, and model-specific wire details remain inside that package. Refer to AI inference servers for deployment and operational guidance.

When given the worker’s ready file, VoiceAgent signals process readiness after its private media transport initializes; DeviceIOHub separately owns client readiness. VoiceAgent also owns hub transport, voice gating, its private media pipeline, signal handling, and cleanup. Its transcript publication follows the active mode and ASR backend’s policy, so only selected final pre-gate transcripts are published separately from gate-accepted user queries. It can also publish typed participant lifecycle events and consumes typed voice output. Join and leave publication stays non-blocking for the media pipeline while preserving order per participant. Application agents subscribe to the events they own and clean up their own participant state. Pipecat and the media session remain implementation details.

Applications may publish selected compact payloads to WEB_EVENT_TOPIC for a WebEventsAgent to display. The viewer owns only a bounded live history and a loopback HTTP listener by default. It is not persistence, does not inspect every runtime topic, and does not enter model, voice, media, or hub authentication paths. The application explicitly selects which typed events to forward.

Several agents may publish speech to the same VoiceAgent. Each declares its output as a reply to a request or an unprompted update, and VoiceAgent owns when each is spoken: it orders replies per request, drops output for superseded requests, lets urgent updates cut current speech, and speaks pending updates together when the bot falls silent, using playback state from the media output. An optional combiner such as llm_update_combiner(llm) rewrites pending updates into one utterance. This policy stays inside the voice consumer so producers never manage stream identifiers or interruption flags.

ProcessorEndpoint is the minimal agent-side hub boundary. It receives data, audio, frame signals, and participant events and sends participant-routed return traffic. Video tools acquire frames on demand; raw pixels and media stay on the hub path rather than entering agent APIs.

Documentation boundary#

Public API membership comes from literal __all__ declarations. Sphinx parses the source without importing SDK packages, then renders co-located docstrings, annotations, and defaults. Package READMEs are concise entry points into these published package guides; cross-call behavior, architecture, and ownership live here and in the package reference pages. Guides, troubleshooting, and migrations remain handwritten because they describe workflows and decisions rather than API shape.