Simple VLM example#

Overview#

Warning

Simple VLM is deprecated. It remains available for the browser demo and existing integrations. Use Conversation example as the starting point for new applications.

This legacy reference shows a single visual-question pipeline, from spoken or typed input through image selection to a streamed answer.

Refer to the quickstart to run the demo.

Composition#

Responses stream to Pocket TTS and the vlm.response data topic.

The orchestrator starts DeviceIOHub and the worker. Passing --session-recording also starts passive session capture; capture is disabled by default. The yaml/models.json profile configures the Cosmos3 Nano and Pocket TTS client adapters and endpoints. yaml/voice_gate.yaml configures the speaker-ASR endpoint. Start those endpoints together with the sample’s model-server stack.

VoiceAgent owns worker readiness, hub transport, voice gating, TTS, signals, and cleanup. It publishes accepted speech and typed text as a participant-scoped UserQuery. SimpleVlmAgent always invokes CurrentFrameTool. The tool uses the hub’s fresh frame when DeviceIOHub is already observing video for that participant. When video is off, the same tool asks StreamKit to capture one image and return it through a targeted LiveKit byte stream. The fallback is invisible to the agent: it receives the same opaque image-reference shape and passes it to StreamingImageQueryTool before publishing response chunks to voice output. Camera bytes remain on the hub path, image locations are redacted from VLM telemetry, and the sample has no MCP path.

The voice query therefore determines when a still is needed. A spoken “what is this?” does not require an always-on video publication, while clients that are already streaming do not receive a redundant capture request.

A newer participant turn cancels the superseded vision request and interrupts its voice response. Participant departure releases the sample agent’s cached frames and tasks. This is a minimal composition for a single foreground streaming image query.

The shared voice layer returns accepted queries as UTF-8 text on voice.user-query; the sample has no display adapter. The web client’s RECEIVED list displays it using the existing text-message handling, for example Speaker 0: What is in front of me?. The number is the diarization model’s label within this participant’s voice connection. Typed input and ordinary STT use User: .... Conversation controls do not produce query messages.

The sample enables speaker.diagnostics: true in yaml/voice_gate.yaml. RECEIVED also shows detections and ignored transcripts on voice.speaker-diagnostic, for example Speaker 1 [ignored] detected and Speaker 1 [ignored]: What time is it?. Before enrollment, detections are marked waiting for enrollment. These display-only messages cannot invoke or interrupt the agent. Set speaker.diagnostics: false to stop displaying diagnostics and stop transcribing non-owner voices after enrollment. Refer to AI inference servers for decoding and resource cost. Ordinary STT, including the NIM path, returns accepted queries without speaker diagnostics.

Source map#

The worker package is under agent-samples/simple-vlm-example/worker/simple_vlm_example_worker/:

File	Responsibility
`__main__.py`	Parses launcher arguments and starts the worker
`app.py`	Composes `VoiceAgent`, `SimpleVlmAgent`, services, and readiness
`agent.py`	Owns participant-scoped vision turns, cancellation, and cleanup
`config.py`	Resolves worker, model, voice-gate, and prompt settings
`prompts/system.txt`	Defines the default VLM instruction

Readiness and warmup#

Before announcing readiness, the worker performs a streaming VLM request with a 1280×720 JPEG and consumes the response. This exercises the production multimodal path so the first user query does not pay its initialization cost. The warmup retries failed inference requests until one succeeds. It does not call a health endpoint first. With the checked-in voice-gate configuration, startup also waits for the required speaker-ASR service. Batch STT and TTS health do not gate startup; start the sample’s model-server stack before the sample so speech requests can succeed.

Configuration#

Run and edit the sample from agent-samples/simple-vlm-example/. The orchestrator always passes yaml/device_io_hub.yaml to DeviceIOHub and yaml/simple_vlm_example_worker.yaml to the worker. The worker resolves its models and voice-gate files relative to the worker YAML, so the checked-in layout works without command-line configuration arguments.

File	Owns
`yaml/simple_vlm_example_worker.yaml`	Frame freshness and wait limits, VAD, idle timeout, and optional prompt overrides
`yaml/voice_gate.yaml`	Conversation controls, optional wake phrases, listening chime, and follow-up window
`yaml/models.json`	Model adapters and shared endpoints
`yaml/device_io_hub.yaml`	LiveKit room and ports, web and token servers, and network behavior
`yaml/media_capture.yaml`	Opt-in media-hub capture, NVENC output, caption layout, and retention
`worker/simple_vlm_example_worker/prompts/system.txt`	Default VLM instruction

Edit the owning file, preserve the field’s YAML type, and restart simple_vlm_example; configuration is loaded only at process startup. For example, lower silero_threshold in the worker YAML if quieter speech is being missed, or change conversation.start_phrase and conversation.stop_phrase in the voice-gate YAML. Set conversation.require_wake_phrase: true to require its configured magic_phrases during an active conversation. Set mode: hybrid to also accept wake-addressed one-off questions while closed, or mode: wake_only to use wake-addressed turns without a persistent conversation. Refer to Conversation controls. Relative paths in the worker YAML are resolved from yaml/, not from the shell’s current directory.

Changing an entry in models.json changes only the client adapter or endpoint that this sample uses. It does not reconfigure or restart the shared server. For a checkpoint, port, GPU, or model-runtime change, refer to Customizing model servers, update the sample model stack, stop that persistent stack, and start it again before restarting this sample.

Refer to the generated configuration reference for exact fields, checked-in values, and adjacent YAML comments.

After connecting, say Hey agent, let's start talking, then ask questions without a wake phrase. Say Hey agent, let's stop talking to finish. These controls work with the sample’s existing STT endpoint, including NIM. Refer to Conversation controls for paused phrases and the legacy wake mode.

Opt-in session capture#

Capture is disabled by default. Run uv run simple_vlm_example --session-recording to start device_io_capture immediately after DeviceIOHub and record normalized hub media without joining the LiveKit room. Each participant connection then creates a bundle under ~/.local/share/xr-ai/captures/simple-vlm-example/ containing a canonical raw bundle plus one derived captioned NVENC H.264 video in a fast-start .mp4 with timestamp-aligned 48 kHz stereo AAC-LC device/agent audio, retained source H.264 and WAV tracks, exact raw audio chunks, frame/audio timestamp indexes, a dedicated transcript, optional frame-linked observations, inbound and outbound data, and a manifest. Text returned on vlm.response appears in the scrolling data panel; final STT and text sent to TTS use the larger primary caption. This sample explicitly selects the demo profile, which invokes the separate capture renderer after the raw participant-lifetime bundle closes.

Encoding and file writes run in the separate capture process behind bounded queues. If recording falls behind, it drops pending capture frames rather than backpressuring the hub or worker. MP4 finalization requires ffmpeg on PATH. Omit --session-recording when a deployment must not retain device media.

For a captioned MP4, disconnect the participant and wait for video/session.mp4 and the manifest’s renderings.captioned_mp4 entry before stopping the stack. Shutdown finalizes active raw bundles without starting new renders. Refer to Server runtime for rendering a completed raw bundle separately.

Wake phrases match at the start of a final transcript or after sentence-final ., ?, or ! punctuation followed by whitespace or a closing quote. Text before that boundary and the phrase itself are discarded before dispatch.

Relay output#

The worker writes relay-events.jsonl beside worker.log in the per-run log directory printed at startup. It records runtime publications, receiving-agent callbacks, the complete simple-vlm.turn lifetime, and nested vision and model calls. Per-token model marks, incremental voice fragments, and empty stream terminators are omitted. Voice and STT each emit one semantic scope for a completed operation. TTS emits one voice.tts scope for each sentence sent to synthesis. Raw audio is represented only by size, duration, and sample rate.

Image locations appear as <redacted:image>. Prompts, questions, responses, participant identities, and correlation metadata remain visible and may be sensitive. Inspect a running sample with:

tail -F /tmp/log_simple-vlm-example_*/relay-events.jsonl