xr-ai-interaction#

Overview#

Your application’s capabilities may need to serve both spoken and typed requests. Shared interaction messages let the same agents answer through either interface and distinguish answers from unsolicited updates, such as a timer finishing or a monitored condition changing.

Messages#

The payload types are UserQuery, ResponseOutput, ParticipantJoined, ParticipantLeft, and Interrupted. The package depends only on Pydantic. Applications choose topic names; the types define only the payloads.

VoiceAgent publishes recognized speech as UserQuery and speaks ResponseOutput from VOICE_OUTPUT_TOPIC. A text front end can publish and render the same messages.

Reply and update output#

A producer declares what kind of output it emits; the consumer decides when it is rendered. ResponseOutput.kind is required:

  • reply answers one user request and carries that request’s request_id. A consumer renders replies for one request in order and drops output for a request that a newer request or an interruption superseded.

  • update is unprompted information, such as a timer or a monitor reading. It has no request_id. The consumer renders updates when the participant is not being spoken to and may combine several into one. urgent=True marks an update that cuts current output and plays next; only updates can be urgent.

  • withdraw carries no content. It asks the consumer to drop the updates the producer queued for the participant that have not started playing, for example when a notice describes a state that has since changed. Output that is already playing and other producers’ updates are kept.

A reply is either one message without response_id, or a stream of fragments that share a response_id, carry final=False, and end with an empty final=True terminator. Updates are always finite. Producers never choose interrupt flags or stream identifiers themselves.

ResponseOutputs frames these messages so agents do not build them by hand. It wraps anything with an async publish(topic, message, *, participant_id=...) method, such as AgentRuntime or the RuntimeContext passed to a subscriber:

from xr_ai_interaction import ResponseOutputs

outputs = ResponseOutputs(ctx, VOICE_OUTPUT_TOPIC)

await outputs.update(participant_id, "The timer is done.")
await outputs.update(participant_id, "The oven is too hot.", urgent=True)
await outputs.withdraw(participant_id)

async with outputs.reply(participant_id, request_id=query_id) as reply:
    async for fragment in model_stream:
        await reply.send(fragment)

Reply publishes each fragment, then an empty terminator when the block exits, including on an exception or cancellation, unless nothing was published or a publish failed. Once its task is being canceled, send publishes no further fragments. Pass closed_errors to suppress the error a closed consumer raises for the terminator, and read reply.delivered afterward to learn whether the terminator was published.

The framing contract on ResponseOutput in the Python reference below describes how consumers assemble streamed fragments. Stream tracking and duplicate handling belong to the consumer. Refer to Voice output and interruption for VoiceAgent’s output policy.

Python API reference#

The generated Python reference documents the exported names and their fields and validation.