Connecting clients

Overview

Connect your agent to a browser, phone, or XR device using the supplied client applications. Use them to test an interaction on your target device or as a starting point for your own client integration.

Client matrix

Client	Directory	Transport	Build
Web	client-samples/web/	LiveKit from CDN	None
Web-XR	client-samples/web-xr/	Local LiveKit and CloudXR bundles	web-xr-build/build.sh
Android	client-samples/android/	LiveKit Android	Android Studio or Gradle
iOS/visionOS	client-samples/ios-visionos/	LiveKit Swift and CloudXRKit	Xcode
Native C++	client-samples/native/	LiveKit C++	CMake

The clients share a StreamKit shape: one transport-neutral StreamSession delegates to a StreamingBackend, and LiveKitBackend is the only layer that imports a LiveKit SDK. Connection, microphone, camera, participant status, data, network metrics, request-driven image capture, and file transfer remain separate operations.

Camera streaming quality

Pass an optional CameraEncodingConfig through CameraConfig.encoding when starting video. The settings belong to the local publishing client. No hub configuration or client-server policy exchange is involved. LiveKit mapping stays inside the backend; other backends apply the settings they support.

Preset	Adaptation preference	Multiple-resolution publication
Detail	Favor resolution; reduce FPS under pressure	Disabled
Motion	Favor FPS; reduce resolution under pressure	Backend default
Balanced	Allow both resolution and FPS to change	Backend default

All three presets start with ceilings of 3 Mbps per encoded stream and 30 FPS. These are configurable policy defaults, not values derived from a survey of glasses or device measurements. They match LiveKit’s 16:9 1080p preset reference; the adaptation preference distinguishes the three presets, not a measured quality or bandwidth guarantee for a particular device. Override the FPS ceiling for workloads that need fewer, more detailed frames. Bitrate is an encoder ceiling, not a total bandwidth cap: simulcast, protocol overhead, and other media can add traffic. Actual FPS and bitrate may be lower. Omitting encoding preserves backend defaults. C++ also retains its existing zero-as-unset numeric fields; its named presets supply the explicit ceilings. The C++ LiveKit SDK exposes the primary encoding limit but not lower simulcast layer controls. With simulcast enabled, that SDK still chooses lower-layer budgets; StreamKit cannot enforce their ordering relative to a custom primary limit. No C++ runtime behavior is changed to hide this limitation. Explicitly disabling simulcast avoids that unsupported combination.

const encoding = new CameraEncodingConfig({
  ...CameraEncodingConfig.detail,
  maxFramerate: 10,
  maxBitrateBps: 3_000_000,
});
await session.startCamera(new CameraConfig({ encoding }));

var encoding = CameraEncodingConfig.detail
encoding.maxFramerate = 10
try await session.startCamera(config: CameraConfig(encoding: encoding))

val encoding = CameraEncodingConfig.DETAIL.copy(maxFramerate = 10)
session.startCamera(CameraConfig(encoding = encoding))

streamkit::CameraConfig camera;
camera.encoding = streamkit::CameraEncodingConfig::Detail();
camera.encoding->max_framerate = 10;
session.StartCamera(camera);

The preference and simulcast switch can also be configured independently. On web, Android, and Apple clients, publishing with an explicit bitrate limit keeps configured lower-layer budgets at or below the primary budget. If the largest SDK lower-layer budget exceeds the requested primary budget, all lower budgets are reduced by the same SDK-derived proportion. This preserves their relative ordering, dimensions, and frame-rate settings without inventing a new layer profile. Lower budgets remain unchanged when no reduction is needed; when reduction is needed, even an already smaller lower-layer budget can decrease. An encoding profile that cannot preserve positive integer budgets and distinct budget ordering is rejected during publication rather than silently rounded into a different profile. This is not an aggregate bandwidth or power-management policy. Favoring detail alone changes encoder adaptation; the Detail preset additionally disables lower simulcast layers. Capture format remains selected by the platform or external frame producer, preserving the camera’s aspect ratio.

For externally supplied Apple or Android frames, pass the same encoding policy as the optional encoding argument to session.injectVideoFrame on the first frame that publishes the track. Later frames keep the active policy. C++ takes the policy through StartCamera before the first FrameSink frame. Stop the camera and start a new publication to change settings. Custom backends using the original frame-injection method retain their existing behavior and ignore the optional encoding policy until they implement it.

Android device cameras use LiveKit’s cached camera track; StreamKit does not dispose that SDK-owned allocation. Injected Android tracks remain explicit allocations. Camera stop during LiveKit reconnection reports not connected: the SDK can retain an unpublished track for republishing, so stop cannot safely release it until reconnection completes or the room disconnects. A reported partial native disposal failure prevents replacement rather than retrying an already partly disposed allocation. Retained camera failures do not prevent transport reconnection for audio or data. An explicit disconnect still reports cleanup failures after closing admission; a failed transport close is retained and is not silently treated as success. Android reclaims SDK-held capture only after actual custody evidence from that room, not merely its disconnected state or an absent publication. It does not wait for an event the SDK may never emit: an unentered room has no terminal obligation, and unknown entered-room capture remains quarantined until an actual receipt arrives. SDK-internal cleanup failures that LiveKit does not expose remain an SDK limitation; a capturer disposal callback is not proof that every private native source was released.

Apple camera transitions use one ordered queue. Every stop cancels earlier queued starts and drains cleanup, including a stop admitted after an intervening start. Repeated stop when already idle performs no capture work; a later explicit start remains a new request. First-frame publication uses the same queue. Failed capture cleanup retains the exact track for a later stop or disconnect attempt, without preventing transport reconnect; input already admitted before or during stop cannot replace it. A terminal room disconnect also attempts that cleanup. Neither backend treats a removed publication as proof of completed capture release.

These settings do not guarantee minimum streaming resolution, minimum FPS, or latency. Under insufficient bandwidth, delivery may stall or lose frames. Consumers must still handle changing frame dimensions. A receiver’s LiveKit quality request is also a preference, not a minimum-resolution guarantee.

Request-driven image capture

Every StreamKit client can opt in to an image-capture handler. When an agent needs a picture for a participant whose video is not already streaming, DeviceIOHub sends a targeted camera.capture.request. The handler captures and encodes one JPEG, PNG, or WebP image; StreamKit returns it to the hub with LiveKit’s chunked byte-stream transport. Requests include an opaque correlation ID and deadline. The web, Android, and Apple SDKs cancel pending asynchronous capture work on disconnect or when a superseding turn expires the request. A client without an installed handler, or whose handler fails, returns an immediate rejection so the requesting tool does not wait for its full timeout.

The web, Android, and Apple sample apps expose one persisted Camera Mode selector. Off installs no image handler and publishes no video, On-demand images installs the handler without publishing video, and Live video publishes the selected camera without installing the image handler. The first-launch default is Off. The selection is not reset by disconnect, reconnect, or app relaunch, so reconnecting restores the selected behavior without silently changing the user’s authorization choice. In on-demand mode, the handler briefly opens the selected camera when necessary and stops it after the still is encoded. Mode changes gate new requests: a request accepted while On-demand images is selected finishes normally and releases its resources, even if the user selects Off before it completes. Off prevents the next request; it is not a retroactive cancellation signal. Disconnect and the request protocol’s cancellation message still cancel pending work.

The native StreamKit API exposes the same callback for the host application’s camera pipeline. The native LiveKit backend requires client-sdk-cpp v1.10.2 or newer. Capture streams are targeted to the configured hub identity and are limited to 8 MiB at the connector. They are not broadcast to peer participants.

This capability is independent of visual inference. An agent calls the existing CurrentFrameTool and can pass its returned ImageReference to a VLM, an image-processing tool, a workflow, or another participant-scoped agent. The tool resolves a fresh hub frame when video is streaming and requests a client capture only when one is not available. This applies to every tool invocation, including periodic monitoring: installing the client capture handler opts that client in to brief camera activation when a caller needs a still while video is off. Agents do not branch on transport or camera state.

External camera sources, including glasses SDKs, use the same mutually exclusive modes at the application boundary. In Live video, pass each SDK frame to injectVideoFrame; that API publishes a video track by design. In On-demand images, keep or request the latest SDK frame in the source adapter and encode it only from onImageCaptureRequested—do not pass it to injectVideoFrame, because that would publish video. In Off, stop or discard the SDK feed and leave the handler unset. This routing is client-side and does not require a new StreamKit API. The Android virtual-camera source demonstrates both paths: live mode injects its generated frames, while on-demand mode encodes one generated frame without publication. Apple integrations apply the same routing to the CMSampleBuffer supplied by the camera SDK.

The same resolution rule can extend to a future short-video selector: resolve a recorded hub window first, then ask a capable client to record and upload a bounded segment when no local window exists. Client video capture is not implemented yet, and no provisional video protocol is exposed.

Client-to-agent files

Every StreamKit client exposes sendBytes and sendFile (SendBytes and SendFile in C++). Both operations require an application topic; in-memory bytes also require a name. MIME type and string attributes are optional. Send completion confirms that the local LiveKit byte stream closed. Applications define their own agent-processing acknowledgement when needed.

await session.sendBytes(pngBytes, {
  topic: 'image.response',
  name: 'capture.png',
  mimeType: 'image/png',
  attributes: { request_id: requestId },
});

Applications can group independent transfers with attributes such as a request ID and index. StreamKit supplies transport framing and emits only complete files, but it does not impose application-specific grouping or timing.

Topics, names, and MIME types are limited to 255 UTF-8 bytes. A transfer may include up to 32 application attributes; keys are limited to 128 bytes, values to 1 KiB, and their combined encoded size to 8 KiB. Every client validates these protocol limits before opening the transfer.

Graphical clients display LiveKit connection quality, round-trip time, and receive jitter; the C++ sample reports them through its callback. Backends sample existing WebRTC statistics about once per second. No hub message or separate telemetry service is involved.

Shared connection flow

The hub prints a URL, room, development token, and web-client URL at startup. Browser, Android, and Apple clients connect to the hub’s web-server port 8080, which proxies LiveKit on the same origin. The native C++ client also supports this TLS path with --secure --port 8080. Its port 7880 default is a plaintext direct-debugging exception; restrict it to the XR AI host or a trusted source subnet as described in Networking and firewall.

A client may paste the printed 24-hour development JWT or fetch a shorter-lived token from:

GET https://<host>:8080/token?identity=<identity>

The response is a JSON object containing token, room, and url. StreamKit token fetchers also accept a plain JWT response. The room is encoded in the token; clients do not send a separate room query parameter.

The default HTTPS server leaf is signed by a development root CA stored under ~/.local/share/xr-ai/. Browsers allow a development click-through; Android and Apple devices install the public root from /cert using the platform trust procedures below. Use a public or managed certificate for production.

Web

The basic page in client-samples/web/ uses plain ES modules and loads LiveKit from jsDelivr. It has no build step. Open https://<host>:8080, accept the development certificate warning, leave Token URL blank to use /token, and connect. The camera preview follows the published track’s aspect ratio.

Web-XR (xr-render-demo)

The Web-XR page uses same-origin bundles under client-samples/web-xr/vendor/. The xr-render orchestrator builds missing bundles automatically when npm is available. To rebuild manually:

cd client-samples/web-xr-build
./build.sh

The idempotent script reads the CloudXR version from .sdk-version, reuses a matching versioned tarball when present, otherwise reuses the matching IsaacTeleop tarball at ~/hub/IsaacTeleop/deps/cloudxr/ when available, and finally downloads the public NGC tarball. It installs npm dependencies, writes both ESM bundles to ../web-xr/vendor/, and records the SDK version beside them. The orchestrator automatically rebuilds missing or stale bundles. To bump CloudXR, edit .sdk-version and re-run the script. To bump LiveKit, edit package.json, remove node_modules, and re-run the script.

Android

The Android app is a Jetpack Compose client with selectable Camera2 devices, microphone capture, participant status, arbitrary data, and network metrics.

Requirements and build

Requirement	Version
Android Studio	Koala 2024.1.1 or newer
JDK	17
Minimum Android	API 24
Target Android	API 34

Open client-samples/android/ in Android Studio, allow Gradle sync to finish, select a device, and run. The checked-in wrapper also supports:

cd client-samples/android
./gradlew assembleDebug

Connect and permissions

Enter the server host, port 8080, a unique identity, and either the printed token or the default token URL. Tap Install hub certificate before the first connection; Android opens the system certificate-install flow.

The app requests microphone permission when starting the microphone and camera permission when opening a physical camera. INTERNET and MODIFY_AUDIO_SETTINGS are install-time permissions. BLUETOOTH_CONNECT is a runtime permission on Android 12 and newer; the current sample declares it but does not prompt for it, so grant Nearby devices in system settings before depending on Bluetooth audio routing.

The current Android backend maps every enabled AudioConfig mode to LiveKit’s default microphone capture. It does not yet select distinct voice-processing, software-processing, or raw DSP settings, so the sample UI does not expose a DSP-mode picker. LiveKit plays remote agent audio automatically. Camera choices come from Camera2 and include front, back, extra built-in lenses, and attached USB cameras when the device exposes them.

The root Android build pins the Netty dependency pulled in by AGP’s test tooling above known vulnerable 4.1 releases. After changing AGP or the version catalog, run ./gradlew verifyNettyPin from client-samples/android/.

The app can run as a flat Android panel on Android XR, but that path is not validated. It does not use Jetpack XR or immersive Android XR APIs, and its Camera2 selector is not a passthrough-camera integration.

iOS/visionOS

The Apple client is a checked-in SwiftUI project and local Swift package. It targets iOS 18 and visionOS 26, requires Xcode 26 with Swift 6.2, and resolves LiveKit Swift and CloudXRKit through Swift Package Manager.

Build

Open client-samples/ios-visionos/StreamKitSample.xcworkspace, select the StreamKitSample scheme and destination, choose a signing team and a bundle ID owned by that team, then build and run. The workspace already includes the app project and local StreamKit package; do not create a replacement Xcode project.

The checked-in bundle ID is com.nvidia.xr-ai-example. Changing it does not change the StreamKitSample display name, but it does select a new UserDefaults domain, so saved settings reset on the first launch after a rename.

The simulators stream StreamKit/Sources/StreamKit/Resources/SimulatorFeed.gif instead of a physical camera. Replace that resource to customize the simulated feed.

On an iOS device the preview follows the live camera aspect ratio. The Vision Pro main-camera track goes directly from ARKit to LiveKit and is not copied into the 2D preview; the LIVE badge, rather than the placeholder preview, indicates capture. Start the immersive space before camera capture on a Vision Pro device.

Vision Pro permissions

Main passthrough camera access on a Vision Pro device requires both the com.apple.developer.arkit.main-camera-access.allow entitlement and the team’s Apple-issued Enterprise.license. Put the non-redistributable license at client-samples/ios-visionos/App/Enterprise.license; the build phase copies it into the application. A missing license leaves audio, data, simulator camera, and the rest of the app usable, but disables device passthrough camera access.

Xcode automatic signing supports development builds. App Store and TestFlight distribution require a manually issued provisioning profile that grants the main-camera entitlement.

The native CloudXR path also declares com.apple.developer.low-latency-streaming, which requires an Apple Developer Program team. A team that cannot provision this entitlement can remove it from client-samples/ios-visionos/App/StreamKitSample.entitlements; native CloudXR remains available with higher latency. Simulators do not require the enterprise camera license.

Connect and trust the certificate

Enter the host and port 8080, then use a pasted token or the default /token endpoint. SessionConfig contains only the participant identity; the token encodes the room. Incoming data callbacks receive (topic, data), and _agent.status is delivered separately through the status callback.

The LiveKit WebSocket requires system trust for the hub certificate. In the app, tap Install hub certificate, allow Safari to download the profile, install it under Settings → General → VPN & Device Management, then enable full trust under Settings → General → About → Certificate Trust Settings. Refer to Troubleshooting for certificate regeneration, SAN mismatch, 401, and media-interruption diagnostics.

When microphone or camera state appears stuck, filter Console.app by category:MediaSession to inspect interruption and recovery events. CoreAudio -50 and FigAudioSession -19224 messages also appear during successful microphone starts and are not failure indicators by themselves.

If microphone startup fails with io.livekit.swift-sdk Code=101 "Timed out" after about five seconds, the LiveKit recording engine did not produce the first buffer. Current builds pre-warm that engine before publishing. After leaving XR, an iOS microphone activation sound can play while capture restarts; this is expected.

Native CloudXR

Native Apple clients require the CloudXR native device profile. Start the demo from the repository root with:

NV_DEVICE_PROFILE=auto-native \
  uv run --project agent-samples/xr-render-demo xr_render_demo

The environment variable overrides the auto-webrtc YAML default without a checkout edit. LiveKit remains on port 8080 while CloudXRKit uses its native transport. Connecting or stopping XR does not disconnect the LiveKit agent session. Closing the immersive space or disconnecting from the hub stops the XR session so no render component is orphaned.

In auto-native mode, CloudXR does not use the web client’s port 48322 WSS proxy and needs no additional certificate installation. CloudXRKit’s native transport provides its own encryption without a user-facing trust prompt. The hub certificate on port 8080 remains required for the LiveKit channel.

Native C++

The C++20 sample ships a working backend for LiveKit client-sdk-cpp v1.10.2 or newer. The SDK root must use its official include/ and lib/ layout.

cd client-samples/native
cmake -S . -B build -DLIVEKIT_SDK_ROOT=/path/to/livekit-cpp-sdk
cmake --build build
./build/bin/streamkit_sample --host 192.168.1.100 --port 8080 \
  --secure --token <jwt>

Install the hub certificate in the native host’s trust store before connecting. Refer to Linux native certificate trust for the Ubuntu and Debian procedure. The command uses the supported TLS proxy on port 8080. Insecure port 7880 is a direct LiveKit debugging path; do not expose it outside the XR AI host or a trusted development network.

Without LIVEKIT_SDK_ROOT, it builds in stub mode and reports a connected state without opening a network session. Build the standalone assertion-based tests with -DSTREAMKIT_BUILD_TESTS=ON and run:

ctest --test-dir build --output-on-failure

Test	Coverage
streamkit_mapping_tests	ConnectionState operations
streamkit_agent_status_tests	Canonical, missing, truncated, and empty _agent.status values
streamkit_frame_sink_tests	Move and span frame-injection overloads
streamkit_audio_sink_tests	Audio parameter forwarding and virtual dispatch
streamkit_session_tests	Mock-backed connection, media, data, status, and disconnection lifecycle
streamkit_livekit_backend_tests	State deduplication, disconnect races, stale metrics, and callback exceptions
streamkit_byte_stream_tests	Byte-stream connection admission and stub or SDK-backed send preconditions

Run one test with CTest, invoke its binary directly, or rerun failures:

ctest --test-dir build -R streamkit_frame_sink_tests --output-on-failure
./build/StreamKit/Tests/StreamKitTests/streamkit_frame_sink_tests
ctest --test-dir build --rerun-failed --output-on-failure

The backend implements connection state, data, _agent.status, network metrics, host-injected audio, and host-injected video. StartAudio() and StartCamera() arm publication; the host opens its devices and pushes PCM or frames through AudioSink and FrameSink. Camera facing and device ID are therefore inert in this backend, while optional encoding settings apply before the first frame creates the video track. Real-time producers must prefer the std::vector<uint8_t>&& frame-injection overload; the span overload copies each frame and can add about 1.4 MB of copying for a typical frame.

Token-URL HTTP fetching is not implemented; pass an inline token or subclass LiveKitBackend::FetchToken. GetRoom() is the LiveKit-specific escape hatch for remote audio rendering or AEC reference capture. Microphone processing presets are not mapped because the C++ SDK exposes no corresponding source controls.

Adding another platform

DeviceIOHub uses standard LiveKit. A new client fetches a token, joins through the hub’s port 8080, publishes microphone and optional camera tracks, plays the remote audio track, and reads data topics such as agent.response. Keep the transport-specific SDK behind a StreamKit backend and reuse the same status, interruption, and participant-routing contracts.