Release migration#

DeviceIOHub rename#

Update services/xr-media-hub/ to services/device-io-hub/, the xr-media-hub distribution to device-io-hub, and the xr_media_hub import and command to device_io_hub. Rename xr_media_hub.yaml to device_io_hub.yaml and XR_MEDIA_HUB_NO_WEB_CLIENT to DEVICE_IO_HUB_NO_WEB_CLIENT. The rename has no compatibility aliases.

Shared model-server sample location#

The shared model-server launcher moved from agent-samples/model-servers/ to model-server-samples/model-servers/. Update scripts and configuration paths that name the old directory. From the repository root, run:

uv run --project model-server-samples/model-servers model_servers

From an agent sample directory, use uv run --project ../../model-server-samples/model-servers model_servers. The model_servers command, model profiles, service ports, and cache locations are unchanged. Run uv sync from the new sample directory to recreate its local environment; an environment tied to the old directory does not need to be moved.

Consumer model readiness#

Workers can report ready without STT, TTS, LLM, or VLM availability. Consumer workers no longer poll model health at startup. VoiceAgent waits only for explicitly supplied probes; it no longer adds STT and TTS probes. Start the shared model-server stack and wait for its launcher to return first. Server wrappers retain model startup and reuse checks.

Simple VLM retains its streaming image warmup, and XR Render retains its LLM warmup, including for hosted LLMs. Tea making retains its RAG capability probe and embedding health checks.

Endpoint readiness and health_path settings still control explicit health() calls, but do not enable automatic worker polling. Out-of-tree applications that require the former behavior can explicitly pass probes={"stt": stt.health, "tts": tts.health} to VoiceAgent.

Operator-visible runtime changes#

  • A VoiceOutput sent with both response_id and final=True now closes that response key even when the response contains only one message. Later output from the same participant and producer with that response_id is dropped, and the voice runtime warns once while the key remains among its 1,024 most recent closures. Use a new identifier for each inbound query, such as ctx.metadata.message_id, or omit response_id for a finite one-message response.

  • DeviceIOHub now waits for the hub to acknowledge shared-memory attachment before connecting the LiveKit room or creating its ready file. Missing segments trigger bounded recreation; incompatible layouts and acknowledgement timeouts fail startup. Check the registration error in the hub logs rather than treating a running process as ready.

  • DeviceIOHub no longer falls back to embedded LiveKit development credentials. Set api_key and api_secret in device_io_hub.yaml, or inject LIVEKIT_API_KEY and LIVEKIT_API_SECRET through the environment.

  • Boolean service settings now require YAML booleans or the strings true, false, yes, no, on, off, 1, or 0 (case-insensitive). Numeric 1/0, null values, and arbitrary strings now fail at startup instead of being interpreted by Python truthiness. This applies to vLLM eager, tool, and scheduling flags, Nemotron-Omni BF16 selection, voice-gate listening_chime, lab-monitoring capture_marker_scans, and tea-workflow complete_on_skip.

  • Return audio is paced before IPC by the built-in voice transport and bounded independently for each participant in DeviceIOHub. return_audio_max_buffer_s defaults to 3 seconds; a custom or faulty producer that exceeds the queued-audio duration limit loses its oldest queued frames. The built-in voice transport requires at least 0.12 to maintain its 120 ms reserve. Increase the value for intentionally bursty custom producers, or decrease it for a tighter memory and latency bound when using a compatible custom producer.

Local speech service#

The local speech service changed from Piper to Pocket TTS with no compatibility alias. Replace the piper_tts model preset, piper_tts_server command, and services/piper-tts/ path with pocket_tts, pocket_tts_server, and services/pocket-tts/. Pocket TTS voice names differ from Piper voice names; the checked-in profiles use the CC0 bill_boerst voice.

Pocket TTS now selects a GPU automatically by default, and the checked-in model-server profiles require CUDA on GPU 0. Set device: cpu for CPU-only execution or cuda_visible_devices to change GPU placement. CUDA warmup runs within startup_timeout_s, so increase that timeout when cold initialization exceeds 600 seconds. The service now resolves the PyPI Torch build instead of the CPU-only index; Linux environments therefore include the CUDA library footprint even when execution falls back to CPU.

Removed SDK compatibility surfaces#

This release removes deprecated SDK aliases and the standalone Pipecat compatibility package. Update out-of-tree code as follows:

Removed surface

Replacement

xr_ai_agent

Import ProcessorEndpoint and IPC types from xr_ai_hub.

BrainProcessor and make_voice_pipeline

Put application behavior in an xr_ai_runtime.Agent subscriber and let xr_ai_voice.VoiceAgent own the voice pipeline.

run_voice_pipeline

Configure VoiceAgent directly and run it with await VoiceAgent.run(runtime); its media session is private.

XRMediaHubTransport

Construct xr_ai_voice.HubVoiceTransport and pass it to VoiceAgent only when another application component must share that existing hub boundary.

VoiceSession

Configure VoiceAgent with STT, TTS, VAD, gating, readiness probes, and closeables, then run it with the shared AgentRuntime. The lower-level media session is no longer public.

VadConfig

Import the unchanged tuning model from xr_ai_voice.

GatedQueryFrame

Subscribe to the application query topic carrying xr_ai_voice.UserQuery.

ParticipantJoinedFrame, ParticipantLeftFrame, and InterruptionFrame

Subscribe to application topics carrying VoiceParticipantJoined, VoiceParticipantLeft, and VoiceInterrupted. Participant identity comes from runtime metadata; voice-gate greetings remain session-owned.

BrainResponseEndFrame

Publish a finite VoiceOutput, or terminate an incremental response with final=True.

VadSttProcessor, VoiceGateProcessor, and StreamingTtsProcessor

Configure VoiceAgent with VadConfig, VoiceGateConfig, and text_topic; pipeline processors are private implementation details.

SttClient and TtsClient

Construct services through xr_ai_models.make_stt and make_tts, or use OpenAICompatSTT and OpenAICompatTTS directly.

http_probe, mcp_probe, and wait_for_services

Pass additional readiness callables through VoiceAgent(probes=...); MCP readiness is no longer part of the voice SDK.

xr_ai_pipecat.audio conversion helpers

Let VoiceAgent own media conversion. If an application truly needs raw hub media, use xr_ai_hub types and own the format conversion.

VoiceAgent.text_transform and text_ignore_topics

VoiceAgent treats only untopiced client data as direct text. Use text_input=False to disable it; transform application queries in their subscribing agent.

xr_ai_models.config, factory, openai_compat, and protocols

Import public names directly from xr_ai_models. This includes KIND_OPENAI_COMPAT, ModelKind, Category, and Spec.

LiveVisionTool and StreamingVisionTool

Select with CurrentFrameTool, then pass its ImageReference to ImageQueryTool or StreamingImageQueryTool.

HistoricalVisionTool

Select with VideoMemoryTools.get_historical_frame, then pass its ImageReference to ImageQueryTool.

Recorded query_video(start_us, end_us) RPC

Use get_historical_video(start_us, duration_seconds) or get_latest_video(duration_seconds). The new query_video name is VLM inference over caller-selected TimedImage values.

get_frame_from_time(reference_time_us, second_ago)

Subtract the offset in the caller and use get_historical_frame(start_us).

HistoricalFrameResult.path and SampledVideoFrame.path

Read the canonical exported-frame location from result.image.uri.

xr_ai_tools.qr_code.QRCodeTool

Initialize xr_ai_tools.marker_tracking.MarkerTrackingTool; select QR and/or ArUco with marker_types, then use the same track_markers request and result contract for either family.

Implicit development credentials from LiveKitConnectorConfig() or LiveKitConnector()

Pass a LiveKitConnectorConfig with explicit api_key and api_secret. The YAML loader also accepts LIVEKIT_API_KEY and LIVEKIT_API_SECRET.

Pipecat remains an internal implementation detail of xr-ai-voice; applications no longer assemble or subclass its frame processors.

The source directories now match their Python imports: agent-sdk/xr-ai-hub-client/ became agent-sdk/xr-ai-hub/ and agent-sdk/xr-ai-agent-runtime/ became agent-sdk/xr-ai-runtime/. Distribution names remain xr-ai-hub-client and xr-ai-agent-runtime, so package dependency names do not change.

If upgrading a checkout that already downloaded model weights, follow the model-cache migration to reuse the ignored caches rather than downloading them again.