Server runtime#

The services/device-io-hub/ package hosts the DeviceIOHub — the single process clients connect to and agents fan out from. It owns the internal LiveKit transport, the shared-memory + ZMQ IPC boundary to agents, the per-participant return path, and the same-origin wss:// proxy that fronts LiveKit signaling.

From the repository root, run it as one process:

uv run --project services/device-io-hub device_io_hub \
  --config services/device-io-hub/device_io_hub.yaml

Configuration comes from a device_io_hub.yaml file. When none is found, non-secret defaults are used, but LiveKit credentials must still come from LIVEKIT_API_KEY and LIVEKIT_API_SECRET. services/device-io-hub/device_io_hub.yaml is the reference copy documenting every field; each sample ships its own copy under its yaml/ directory. Relative paths inside the YAML (such as web_client_dir) resolve against the YAML file’s own directory, not the working directory.

For where the hub sits in the wider system, refer to Architecture.

DeviceIOHub#

The hub is one hub, many clients, many agents. A single instance fans the inbound media stream out to every connected agent and routes any return traffic back to the originating client only.

On startup, __main__.py loads the configuration, constructs a HubEndpoint (the IPC server), registers hub-local callbacks (on_frame, on_audio, on_data, on_participant), and starts the hub receive loop before bringing up the LiveKitConnector. The connector waits for the hub to acknowledge shared-memory attachment and layout validation before connecting the LiveKit room. The ready file is created only after connector startup and the remaining startup configuration succeed.

The hub task and connector task run concurrently, and the process waits for SIGINT or SIGTERM to shut down. Startup failures attempt cleanup of every owned component. Cleanup errors are logged without replacing the original startup failure. A periodic stats loop logs per-participant video, audio, and data rates.

Isolation contract#

The hub is not a routing switch between participants. There is no supported path for participant A’s media or data to reach participant B. The only supported flow is:

participant → hub → consumer (agent) → hub → same participant

This is enforced at several layers:

  • send_return_audio, send_return_data, and send_return_audio_flush validate that the target participant is currently connected; messages for unknown participants are dropped with a warning.

  • Return-traffic topics (return_audio.*, return_audio_flush.*, return_data.*) are transport-infrastructure-only. Connectors consume them for delivery and the optional capture process observes them read-only; an agent’s default subscription excludes them.

  • On the LiveKit side, return audio is published as one track per participant with subscribe permissions restricted to that participant, and return data is addressed with destination_identities (refer to the per-participant return path).

Note

This isolation is a property of the hub’s routing, not a limitation of the transport. LiveKit natively supports client-to-client communication, and an application is free to use those native features directly for peer-to-peer media or data. Doing so is outside the scope of XR AI: the hub neither routes nor guarantees that traffic, and it is not portable across transports. Build on the hub’s participant ↔ agent contract for behavior that must port across backends.

Internal LiveKit transport#

LiveKit is an internal transport implementation detail. It is not exposed to agent APIs: agents speak only the IPC protocol below and never need to know which transport carries the media.

LiveKitConnector (transport/livekit/) owns the transport lifecycle:

  1. Starts the LiveKit server in a host-networked Docker container (plaintext signaling on port 7880, plus WebRTC TCP/UDP media ports 7881/7882). The generated configuration does not restrict the signaling listener to loopback, so deployment firewalls must control direct access.

  2. Optionally starts the browser-facing web server and/or token server.

  3. Creates its shared-memory ring and registers itself as a ConnectorEndpoint with the IPC layer, waiting for the hub acknowledgement before proceeding.

  4. Connects a Python RoomClient to the LiveKit room. The room client is subscribe-only — it never publishes media of its own except per-participant return-audio tracks.

The connector translates LiveKit room events into IPC messages: it pushes decoded frames, audio chunks, and data into the hub, and emits participant join and leave events.

Note

The LiveKit connector requires NVENC and NVDEC hardware video codecs, which it checks at startup.

IPC boundary to agents#

The hub and its producers and consumers communicate over ZMQ using msgpack-encoded messages. The layer lives in services/device-io-hub/device_io_hub/ipc/ and defines three endpoints:

Endpoint

Role

Who

ConnectorEndpoint

producer + return-traffic receiver

LiveKit connector process

HubEndpoint

server: dispatch + fan-out

DeviceIOHub process

ProcessorEndpoint

subscriber + publisher

agents, analytics, downstream processors

The hub binds two sockets (defaults shown):

  • PULL on ipc:///tmp/xr_hub_in — connectors PUSH inbound media here.

  • PUB on ipc:///tmp/xr_hub_pub — consumers SUB here for the fanned-out stream.

connector_A ──PUSH──┐
connector_B ──PUSH──┤─► PULL   HubEndpoint   PUB ──SUB──► consumers (agents)
connector_N ──PUSH──┘    ↓ dispatch
                      on_frame, on_audio, on_data, on_participant

Each connector owns and creates its own shared-memory ring buffer and announces it to the hub with a ConnectorRegistration message; the hub opens and validates that buffer before acknowledging registration. Duplicate registrations for the same connector and segment are acknowledged without replacing the mapping. Video frames travel zero-copy through the ring buffer: the connector writes pixels into a slot and pushes a lightweight FRAME_SIGNAL (metadata) at full frame rate; consumers that want the pixels issue a FRAME_REQUEST, and the hub replies with the held slot’s FRAME_DATA. Audio, data, and participant events are carried inline as msgpack payloads.

Messages are tagged with a MsgType and routed by topic. Topics follow the "<type>.<participant_id>.<track_or_topic>" convention, and ZMQ’s byte-prefix subscription lets a consumer subscribe at any granularity:

audio                    — all audio, all participants
audio.alice              — all of alice's audio tracks
audio.alice.TR_mic_001   — alice's specific mic track
data.alice.chat          — alice's "chat" data channel only
participant              — join and leave events

The message types and codec are extensible: new MsgType IDs can be registered at import time via register_encoder and register_decoder.

Note

Import the IPC types and ProcessorEndpoint in agent code from xr_ai_hub directly, not from device_io_hub.ipc. The agent SDK’s only runtime dependencies are pyzmq and msgpack — importing from the agent SDK avoids pulling in the full DeviceIOHub dependency tree (LiveKit, FastAPI, uvicorn, GPU codecs). device_io_hub.ipc re-exports the same names for the server side.

Media-hub session capture#

device_io_capture is an optional process from the DeviceIOHub package. It sits downstream of HubEndpoint, not inside the LiveKit connector:

device transport → connector → HubEndpoint → agents
                                  └────────→ capture
                         agent return media ─┘

This boundary is transport-independent. Incoming camera, microphone, and data have already been decoded, timestamped, and tagged with participant and track identities. The capture process uses a normal ProcessorEndpoint for that stream and a private read-only subscription to the hub’s routed return topics for agent audio and data. It does not expose return traffic to agent APIs.

Run the hub first, then start capture from the repository root:

uv run --project services/device-io-hub device_io_capture \
  --config services/device-io-hub/media_capture.yaml

Capture is split into a raw timestamped session writer and a derived capture renderer. The writer owns participant/session boundaries, NVENC encoding, raw float32 audio, the aligned stereo WAV, transcripts, observations, event records, retention, and the manifest. Raw capture is the default. The configured profile controls only whether rendering follows capture:

  • raw writes the canonical camera H.264, audio, transcript, event, and timing artifacts and performs no presentation rendering. It does not require FFmpeg.

  • demo writes the same raw bundle, then invokes CaptureRenderer after the session closes. The renderer adds caption and data panels and produces a fast-start MP4 with H.264 video and 48 kHz stereo AAC-LC audio.

The canonical video/session.264 is always encoded from unmodified camera pixels. video/packets.jsonl records each encoded packet’s byte range, source track, key-frame state, and exact timestamp. The renderer consumes those raw artifacts and the transcript/event timeline, performs a separate NVDEC/NVENC pass for its burned-in panels, and records the derived result under renderings.captioned_mp4 in the manifest. It never replaces the raw media.

Render or re-render any completed bundle independently of the capture process:

uv run --project services/device-io-hub device_io_capture_render \
  ~/.local/share/xr-ai/captures/<session-directory>

session_mode: participant records from participant join to leave. session_mode: explicit creates bundles only between reserved agent start and stop messages. A background agent wrapper initializes the private controller with a destination namespace relative to out_dir and fixed capture metadata; the semantic actions exposed to the model can therefore remain parameterless start recording and stop recording. Each accepted start creates a new timestamped child directory in that namespace. The model never selects a filesystem path. These messages remain an internal adapter seam rather than a public agent tool API, and the wrapper can publish frame-linked observations without changing the storage engine.

The SDK implementation is xr_ai_tools.capture.CaptureTools. Applications construct it with the fixed target and metadata, then call participant_tools(participant_id) to obtain the two participant-bound native tools.

Every bundle uses one Unix-microsecond clock and records both absolute and session-relative timing. video/frames.jsonl indexes source sequence, timestamp, track, pixel format, and source/encoded dimensions. video/packets.jsonl retains the encoded-packet byte ranges and timing needed for deterministic derived video. audio/chunks.jsonl indexes direction, sample format, byte range, duration, and timestamps. transcript.jsonl contains final STT and spoken TTS text, and observations.jsonl accepts derived records linked to an exact frame timestamp. events.jsonl remains the complete directional data timeline, and manifest.json is the machine-readable entry point. Its complete and incomplete_reason fields distinguish a normally drained session from a bounded recovery when the return-traffic departure marker is lost.

In a captioned rendering, final STT and the text actually sent to TTS appear as the large primary caption below the sensor image. Each TTS sentence travels through the paced media queue, so its caption is timestamped from the first corresponding audio chunk rather than from synthesis completion. Every UTF-8 data-channel message, inbound or outbound, scrolls through a smaller right-side panel with its direction and topic. Binary data is not rendered, but every data payload remains in events.jsonl. Both panels sit outside the sensor image, so composition never replaces camera pixels. Reserved speech-caption metadata stays inside hub IPC and is never forwarded to the client data channel.

Capture frame requests are coalesced in a bounded queue, and NVENC work runs in dedicated threads in the capture process. Recorder overload therefore drops capture frames without delaying the hub’s publish path. PyNvVideoCodec receives contiguous NV12 CPU input and emits H.264 Annex B chunks with repeated parameter sets and no B-frames. At session finalization, each raw stream’s chunks are joined in timestamp order. When requested, the renderer reconstructs captions from the bundle timeline. FFmpeg preserves the temporary composed H.264 stream, converts the aligned PCM mix to AAC-LC with an explicit 48 kHz stereo layout, preserves recorded frame timing across sparse or dropped frames, resets trimmed audio to the same zero-based timeline, and writes one .mp4 with fast-start metadata. Resolution or LiveKit track changes therefore do not create additional playable outputs. This does not affect the live path.

CaptureRenderer and the demo profile require an ffmpeg executable on PATH with the native AAC encoder. They validate that requirement before rendering so a session cannot silently fall back to an MP4-incompatible PCM or MP3 track. The raw profile has no MP4 projection and therefore no FFmpeg requirement.

Per-participant return path#

Agents send audio, data, and flush signals back toward a specific participant through the same IPC channel. The hub guards every return path by participant id:

  • send_return_audio publishes on topic return_audio.{pid}., dropping the chunk if {pid} is not connected.

  • send_return_data publishes on return_data.{pid}.{topic}, with the same connectivity guard.

  • send_return_audio_flush publishes on return_audio_flush.{pid}. so a processor can cleanly interrupt the agent’s own audio playback.

The trailing . after the participant id terminates the pid segment so that a subscription for alice does not byte-prefix-match a topic addressed to alice2; the connector subscribes with the identical delimiter when a participant joins, and unsubscribes when they leave.

On the LiveKit side the return path maps to per-participant resources: the room client lazily publishes one xr-hub-return-{pid} audio track per participant and refreshes subscribe permissions so each participant may subscribe only to their own return track. Return data is sent with destination_identities set to the target participant, so it is never broadcast to peers. Return audio is paced before IPC by the built-in voice transport, then fed into LiveKit by a per-participant pipe that a flush can drain to interrupt playback. return_audio_max_buffer_s (3 seconds by default) is also a hard per-participant duration bound: if a custom or faulty producer runs ahead of playback, the oldest queued frames are dropped without affecting any other participant. Set this value to at least 0.12 when using the built-in voice transport, which maintains a 120 ms reserve. Smaller values remain available for custom producers whose chunk size and pacing fit within the configured bound.

Agent status aggregation#

_agent.status is the one exception to straight-through return data. The hub does not forward an agent’s status to the client — it records it per (agent_id, participant_id) and publishes the aggregate, taking the least available state across the agents responsible for that participant: loading > processing > idle > ready. Agents that opt into readiness announce themselves with AGENT_PRESENCE when their receive loop starts and detach when it stops, so an agent that is still loading holds the room at loading rather than being masked by a peer that is already ready.

AGENT_PRESENCE carries the agent’s scope — the participants it answers for, or None for all of them. A participant with no responsible agent reads loading. Passive processors never register, and an agent scoped to one pid is excluded from every other pid’s aggregate. Repeat aggregates are suppressed, so the agents’ periodic re-announcements do not become per-agent client traffic.

A status payload without an agent_id comes from an SDK predating aggregation and is forwarded verbatim.

The hub also answers SUBSCRIPTION_PROBE by echoing the token on _probe.{token}. Subscription commands from one socket are applied in order, so the echo tells a processor that its pending SUBSCRIBEs are live — that is what keeps a client from being told ready before its traffic can reach the agent.

Same-origin wss proxy#

The LiveKit server runs plaintext ws:// on port 7880. Its host-networked container and generated configuration do not limit the listener to loopback, so it can be reachable through host interfaces unless a firewall blocks it. Browser, web-xr, Android, iOS, and visionOS clients connect through the same-origin wss:// URL exposed by the hub’s web server. The native C++ client may connect directly to port 7880 only as a local or source-restricted trusted network debugging path. Refer to Networking and firewall.

When web_server_tls is enabled (the default), the web server (_web_server.py) terminates TLS on web_server_port (8080 by default) and mounts a /rtc route that proxies LiveKit signaling bidirectionally to the internal ws://127.0.0.1:7880 (_lk_proxy.py). A stable development root CA and a signed CA:FALSE server leaf are auto-generated on first run (SAN coverage and rotation: Networking); supply cert_file and key_file to use your own. The proxy forwards end-to-end headers so SDK authentication (such as the LiveKit Swift SDK’s Authorization: Bearer) reaches the server, and handles both the versioned (/rtc/v1) and legacy (/rtc) signaling paths.

The web server’s /token endpoint returns a signed LiveKit JWT together with the client connection URL. With TLS on, that URL is the same-origin wss://<host>:<web_server_port> — so the client SDK never needs a per-deployment toggle. A /cert endpoint serves only the auto-generated public root CA as an installable client profile; neither private key is exposed.

Set web_server_tls: false for the two cases where the hub does not terminate TLS itself: a TLS-terminating reverse proxy (nginx, Caddy, Cloudflare Tunnel) sits in front and speaks plain http:// + ws:// to the hub, or localhost-only development where browsers already grant camera and microphone access on http://localhost. Bind the hub to loopback with web_server_host: 127.0.0.1, or source-restrict it with a firewall, when a reverse proxy is the public entry point. In this mode /token returns the direct ws://<request-host>:<lk_port_ws> URL, not the hub’s /rtc proxy. Custom clients behind the TLS terminator must use the external same-origin wss:// proxy URL and must not expose port 7880 publicly.

For runtime symptoms and fixes, refer to Troubleshooting.