xr_ai_voice#

Public voice runtime for XR agents.

Applications configure and register VoiceAgent, and may pass it an update combiner such as llm_update_combiner(); media sessions, audio framing, and pipeline processors are implementation details.

Attributes#

`VOICE_OUTPUT_TOPIC`	Finite or incremental responses consumed by `VoiceAgent`.
`VOICE_TRANSCRIPT_TOPIC`	Non-empty final STT results queued for publication before gate filtering.

Exceptions#

VoiceStreamClosedError

Raised when output targets a voice stream the consumer already closed.

Classes#

`VadConfig`	Utterance endpointing and Silero-VAD tuning for `VoiceAgent`.
`VoiceAgent`	Own voice media lifecycle and bridge runtime input and output topics.
`VoiceTranscript`	One final STT result emitted before voice-gate filtering.
`HubVoiceTransport`	Own the hub endpoint and bidirectional voice media transport.

Functions#

llm_update_combiner(...)

Return a VoiceAgent combiner that rewrites pending updates with llm.

Package Contents#

xr_ai_voice.llm_update_combiner( llm: xr_ai_models.LLMService, *, prompt: str = _DEFAULT_PROMPT, max_tokens: int = 192, timeout_s: float = 5.0, ) → collections.abc.Callable[[collections.abc.Sequence[str]], collections.abc.Awaitable[str]]#

Return a VoiceAgent combiner that rewrites pending updates with llm.

The combiner rewrites several updates into one spoken utterance. It uses temperature 0 with thinking disabled. It falls back to joining the updates with spaces, and logs why, when the model service fails, the call times out, the model returns no text or tool calls, or the rewrite stops at the token limit. A single update is returned unchanged without a model call.

Parameters:

llm – Model service that rewrites the updates.
prompt – System prompt for the rewrite.
max_tokens – Token limit for the rewrite.
timeout_s – Deadline for one rewrite call, in seconds.

Raises:

ValueError – If prompt is empty or max_tokens or timeout_s is not positive.

class xr_ai_voice.VadConfig#

Utterance endpointing and Silero-VAD tuning for VoiceAgent.

silence_duration controls finalization for either batch or speaker ASR. The other fields apply only to the batch-STT Silero path and mirror xr_ai_vad.VadDetector.

stop_probe_after_s — cadence in seconds for up to three STT probes of the partial audio buffer. This gives STOP commands a fast interruption path without replacing final transcription, and lets a configured wake phrase be acknowledged before the utterance ends. A phrase-gated utterance can make up to three partial STT requests plus the final request. Set to 0 or negative to disable probes.

silence_duration: float = 0.8#

Seconds of silence that finalize an utterance with either ASR backend.

Speaker ASR measures inactivity of the selected speaker and checks it at model-chunk boundaries; this is a threshold, not a response-latency bound.

min_speech: float = 0.15#: Minimum speech duration accepted as an utterance.

silero_threshold: float = 0.5#: Silero VAD speech-probability threshold.

stop_probe_after_s: float = 0.25#: Cadence for bounded early wake and STOP transcription probes.

xr_ai_voice.VOICE_OUTPUT_TOPIC#: Finite or incremental responses consumed by VoiceAgent.

xr_ai_voice.VOICE_TRANSCRIPT_TOPIC#

Non-empty final STT results queued for publication before gate filtering.

Delivery is best effort: the bounded queue drops its oldest pending item on overflow, and shutdown discards queued items that have not been published.

class xr_ai_voice.VoiceAgent( *, query_topic: xr_ai_runtime.Topic[xr_ai_interaction.UserQuery], stt: xr_ai_models.STTService | None, tts: xr_ai_models.TTSService, vad: xr_ai_voice.VadConfig, voice_gate: xr_ai_voicegate.VoiceGateConfig, probes: collections.abc.Mapping[str, collections.abc.Callable[[], collections.abc.Awaitable[bool]]] | None = None, ready_file: pathlib.Path | None = None, closeables: collections.abc.Iterable[Any] = (), text_topic: str = 'agent.response', idle_timeout_secs: float | None = None, transport: xr_ai_voice.HubVoiceTransport | None = None, response_capacity: int = 32, text_input: bool = True, participant_joined_topic: xr_ai_runtime.Topic[xr_ai_interaction.ParticipantJoined] | None = None, participant_left_topic: xr_ai_runtime.Topic[xr_ai_interaction.ParticipantLeft] | None = None, interrupted_topic: xr_ai_runtime.Topic[xr_ai_interaction.Interrupted] | None = None, interrupt_on_supersede: bool = False, combiner: collections.abc.Callable[[collections.abc.Sequence[str]], collections.abc.Awaitable[str]] | None = None, stream_idle_timeout_s: float = 15.0, )#

Own voice media lifecycle and bridge runtime input and output topics.

Startup waits for explicitly supplied probes, such as an inference warmup or an application capability dependency, and required speaker ASR. Batch STT and TTS health are not polled automatically. Readiness reports initialized application transport after those startup requirements, not availability of every model endpoint.

voice_gate selects speech through its speaker settings. stt=None is accepted only with enabled, required speaker ASR; startup then waits for that service. Ordinary STT and automatic speaker selection require a batch STTService. Missing that dependency raises ValueError before VoiceAgent allocates media. An injected transport is already allocated by its caller, who retains cleanup responsibility if construction fails.

from_models() resolves that dependency from an already-loaded model profile and the same voice-gate configuration, without application-side backend inspection. The normal constructor accepts explicitly owned clients.

vad.silence_duration owns utterance finalization for either ASR backend, overriding any speaker.silence_duration in voice_gate for this agent. Each agent copies its effective speaker settings; the shared voice-gate configuration is unchanged. Other VadConfig fields tune batch STT only.

Accepted queries also return participant-routed UTF-8 display text on voice.user-query. Speaker ASR includes its session-local speaker label; ordinary STT and typed input use User. This display does not change UserQuery or require an application subscriber. Delivery uses the existing hub transport and retains its backpressure and best-effort semantics.

Each participant has one voice floor. A reply answers one request and supersedes speech for any other request, which is dropped; it waits for a playing update. An urgent update cuts current speech and plays next. Other updates wait until neither the bot nor the participant is speaking and are spoken as one utterance, joined with spaces or rewritten by combiner, an async callable from the pending update texts to the spoken text; a failing combiner falls back to the join. A new user query or a barge-in cancels replies but keeps pending updates; speech that was cut is never replayed. A streamed reply that receives no fragment for stream_idle_timeout_s is closed so it cannot hold the floor.

classmethod from_models( *, models: xr_ai_models.ModelsConfig, query_topic: xr_ai_runtime.Topic[xr_ai_interaction.UserQuery], tts: xr_ai_models.TTSService, vad: xr_ai_voice.VadConfig, voice_gate: xr_ai_voicegate.VoiceGateConfig, probes: collections.abc.Mapping[str, collections.abc.Callable[[], collections.abc.Awaitable[bool]]] | None = None, ready_file: pathlib.Path | None = None, closeables: collections.abc.Iterable[Any] = (), text_topic: str = 'agent.response', idle_timeout_secs: float | None = None, transport: xr_ai_voice.HubVoiceTransport | None = None, response_capacity: int = 32, text_input: bool = True, participant_joined_topic: xr_ai_runtime.Topic[xr_ai_interaction.ParticipantJoined] | None = None, participant_left_topic: xr_ai_runtime.Topic[xr_ai_interaction.ParticipantLeft] | None = None, interrupted_topic: xr_ai_runtime.Topic[xr_ai_interaction.Interrupted] | None = None, interrupt_on_supersede: bool = False, combiner: collections.abc.Callable[[collections.abc.Sequence[str]], collections.abc.Awaitable[str]] | None = None, stream_idle_timeout_s: float = 15.0, ) → VoiceAgent#

Construct voice input from loaded models and voice-gate settings.

Enabled, required speaker ASR uses the configured streaming endpoint without constructing a batch client. Disabled or automatic speaker selection constructs the existing stt model role. Automatic selection retains that real batch client for its startup fallback. Other arguments and resource ownership match the normal constructor.

Raises KeyError when the selected batch stt role is absent, TypeError when it names another model role, and ValueError for an unsupported adapter, non-positive response capacity, or non-positive stream idle timeout. Caller-created clients and transports remain caller-owned when construction fails.

async run( runtime: xr_ai_runtime.AgentRuntime, *, source: str = 'voice', ) → None#: Run the owned voice session and bridge it to a running runtime.

async output( output: xr_ai_interaction.ResponseOutput, ctx: xr_ai_runtime.RuntimeContext, ) → None#: Send one voice message using participant and producer metadata.

exception xr_ai_voice.VoiceStreamClosedError#: Raised when output targets a voice stream the consumer already closed.

class xr_ai_voice.VoiceTranscript#

One final STT result emitted before voice-gate filtering.

text: str#: Final text returned by STT before voice-gate filtering.

timestamp_us: int#: Speech presentation time as Unix microseconds.

class xr_ai_voice.HubVoiceTransport( input_name: str | None = None, output_name: str | None = None, )#

Own the hub endpoint and bidirectional voice media transport.

property endpoint: xr_ai_hub.ProcessorEndpoint#: Return the owned hub processor endpoint.

async wait_until_started() → None#: Wait until the input transport has started its hub IPC receiver.

async send_return_data(msg: xr_ai_hub.DataMessage) → None#: Send participant-routed data through the hub.

property target_participant: str#: Return the participant currently selected for output.

set_target_participant(pid: str) → None#: Select the participant that receives subsequent output.

cleanup_participant(pid: str) → None#: Clear output routing when the selected participant leaves.

shutdown() → None#: Stop and close the owned hub endpoint.