Out-of-Band Teleop Control#
The OOB (out-of-band) teleop control hub lets you coordinate Isaac Teleop from outside the headset — read streaming metrics, inspect connected clients, and push configuration changes — over the same TLS port as the CloudXR proxy.
The hub shares the proxy TLS port (default 48322, override with
PROXY_PORT).
Quick start#
Step 1 — Start the streaming host with OOB enabled
On first use, it is recommended to run once without --setup-oob to
confirm adb devices sees the headset, verify USB debugging is enabled, and
accept the self-signed certificate in the headset browser manually (both the
web client page and the https://<host>:48322 proxy page). Once that
baseline works, add --setup-oob to automate the full flow.
Launch the CloudXR runtime with the --setup-oob flag (add --accept-eula
on first run):
python -m isaaccapture.cloudxr.service run --accept-eula --setup-oob
This will:
Check host prerequisites and start the WSS proxy with the OOB control hub
Wait for one ready headset, even if none is attached at startup
Open the teleop page on the headset via
adb shell am startAccept the self-signed certificate and click CONNECT automatically via Chrome DevTools Protocol (CDP)
Ordinary headset absence, an offline or unauthorized transport, and
browser recovery failures leave the host running with a degraded OOB status.
The lifecycle retries for 60 seconds at a 5-second cadence, then keeps
observing every 5 seconds. A meaningful device, network, rule, or browser
change opens a fresh 60-second recovery episode. Override the positive,
finite values with TELEOP_OOB_RECOVERY_TIMEOUT_SEC and
TELEOP_OOB_RETRY_INTERVAL_SEC; invalid values fail host preflight.
To start the hub without any adb interaction — useful in containers,
CI, or wireless-only environments — set TELEOP_OOB_HUB_ONLY=1 and open
the teleop URL on the headset manually. This mode supports WiFi setup only
and is not compatible with --usb-local:
TELEOP_OOB_HUB_ONLY=1 python -m isaaccapture.cloudxr.service run --accept-eula --setup-oob
You should see output confirming the hub is running:
CloudXR WSS proxy: running, log file: /home/<user>/.cloudxr/logs/wss.2026-04-13T202133Z.log
oob: enabled (hub + USB adb automation — see OOB TELEOP block)
Note
The headset must be:
Connected via USB cable for adb commands (opening the teleop URL)
Connected to WiFi on the same network as the streaming host (for web page access and CloudXR streaming)
Streaming and web page access use WiFi, not USB tethering.
adb forward is used only temporarily for CDP automation.
Step 2 — (Manual fallback) Open the web client on the headset
If the adb automation fails (e.g. headset not paired), you can manually open
the client URL on the headset browser with all three required query
parameters — oobEnable, serverIP, and port:
https://nvidia.github.io/IsaacCapture/client/main/?oobEnable=1&serverIP=<HOST_IP>&port=48322
Replace <HOST_IP> with the streaming host’s LAN IP. The port must
match the proxy port (default 48322).
Note
All three parameters are required. If serverIP or port is missing,
the OOB control channel is silently skipped — the client will still work for
streaming but will not register with the hub or report metrics.
Step 3 — Verify the headset registered with the hub
From a PC on the same network, query the hub state API (-k skips the
self-signed certificate check):
curl -k https://<HOST_IP>:48322/api/oob/v1/state
You should see the headset listed under "headsets" with
"connected": true:
{
"updatedAt": 1776112022900,
"configVersion": 0,
"config": {"serverIP": "<HOST_IP>", "port": 48322},
"headsets": [
{
"clientId": "193f3758-281e-4292-8c36-6541b58963ef",
"connected": true,
"deviceLabel": null,
"registeredAt": 1776112022805,
"lastSeenAt": null,
"lastMetricsAt": null,
"metricsByCadence": {}
}
],
"lifecycle": {"health": "browser_ready", "state": "ACTIVE"}
}
If "headsets" is empty, double-check that the URL on the headset includes
both serverIP and port and that the headset can reach the host over the
network.
Step 4 — (Optional) Push config to the headset
Before or after the headset connects to the CloudXR stream, you can push configuration overrides via the HTTP config API:
curl -k "https://<HOST_IP>:48322/api/oob/v1/config?serverIP=<HOST_IP>&port=48322&codec=av1"
See GET /api/oob/v1/config below for all supported keys.
Step 5 — Stream and poll for metrics
With --setup-oob, CONNECT is clicked automatically via CDP. If running
without it, press CONNECT on the headset manually. Once streaming begins,
the headset reports metrics to the hub every 500 ms. Poll the state endpoint
from a PC to collect them:
# Poll every 2 seconds (adjust to taste)
watch -n 2 'curl -sk https://<HOST_IP>:48322/api/oob/v1/state | python3 -m json.tool'
The metricsByCadence field on each headset entry will now contain live streaming metrics.
ADB automation#
The --setup-oob flag automates headset setup via USB adb:
adb devices selects one ready headset and pins its serial for the service lifetime
am start opens the teleop bookmark URL in the headset browser with the correct
oobEnable=1,serverIP, andportparametersCDP connect forwards the browser’s DevTools socket over
adb, accepts the self-signed certificate interstitial, and clicks CONNECT via Chrome DevTools Protocol (Input.dispatchMouseEvent)
Streaming and web page access use WiFi, not USB tethering. The headset
reaches the streaming host directly over WiFi. adb forward is used only
temporarily during CDP automation to reach the browser’s DevTools socket.
Prerequisites:
adbmust be onPATH(Android SDK Platform Tools)The headset must be connected via USB with USB debugging enabled
The headset must be on the same WiFi network as the streaming host
If any device step fails, the hub keeps serving and retries. Fall back to
chrome://inspect/#devices from the PC or tap CONNECT on the headset
directly. To re-open the page later without restarting the launcher, pass the
bookmark URL above to python -m isaaccapture.cloudxr.webclient (see
CloudXR Service).
Architecture#
Role |
Software |
What it does |
|---|---|---|
XR headset |
Isaac Teleop web client in the device browser |
Registers with the hub via WebSocket, reports streaming metrics periodically (default every 500 ms), receives config pushes. |
Streaming host |
|
Runs CloudXR runtime + WSS proxy + OOB hub on a single TLS port. Opens the teleop page and clicks CONNECT via USB adb + CDP. |
Operator / scripts |
|
Reads state via HTTP, optionally pushes config via HTTP. |
WebSocket protocol#
Endpoint: wss://<host>:<port>/oob/v1/ws
All messages are JSON text frames with {"type": ..., "payload": ...}.
Registration (first message)#
{
"type": "register",
"payload": {
"role": "headset",
"deviceLabel": "Quest 3",
"token": "<optional CONTROL_TOKEN>"
}
}
role must be "headset". The hub replies with:
{
"type": "hello",
"payload": {
"clientId": "<uuid>",
"configVersion": 0,
"config": {"serverIP": "...", "port": 48322}
}
}
Headset → hub: clientMetrics#
{
"type": "clientMetrics",
"payload": {
"t": 1712800000000,
"cadence": "frame",
"metrics": {
"streaming.framerate": 72.0,
"render.pose_to_render_time": 18.5
}
}
}
One message is sent per cadence per tick, carrying the last known value of every metric
reported so far in that cadence. Metric names are the CloudXR.js MetricsName values,
so they match the SDK documentation. The hub stores them as an arbitrary
{name: value} map per cadence, so metrics added by a future SDK need no hub change.
Browser health exchange#
After each automated CONNECT, the hub sends
healthProbe {probeId, lifecycleGeneration} to a newly registered
browser page. That page answers
healthReport {probeId, lifecycleGeneration, pageTimestamp, streamStatus,
streamPhase, terminalEventId, terminalDetail, lastMetricsAt, metricCadences}.
streamPhase is one of idle, connecting, retrying, streaming,
or terminal. A terminal event ID remains stable while that failure is
replayed and changes only for a later, distinct terminal failure. Older clients
that send only the boolean streamStatus remain supported. The hub accepts
only the matching probe and generation from the target connection. It uses its
own receive time for freshness; the headset clock is diagnostic only. A
matching reply means the UI, WSS, and bidirectional browser control path are
ready. The lifecycle
reports browser_ready while waiting for a stream. In USB-local relay
mode, active additionally requires streamStatus=true and fresh
post-CONNECT client metrics; coturn listening and ADB rules alone are only
TURN prerequisites.
If CONNECT was dispatched but a fresh browser report has not arrived, the
lifecycle remains degraded in VERIFYING_BROWSER and continues probing
without clicking CONNECT again. After a USB cable replug, the lifecycle enters
VERIFYING_EXISTING_BROWSER and gives the client’s configured reconnect loop
time to finish. Fresh post-repair stream metrics return it to ACTIVE without
closing, reloading, or navigating the surviving tab. One terminal event permits
one same-tab CONNECT click; a full browser launch is reserved for the case where
no usable tab survives the bounded grace period. The lifecycle is the sole
owner of CDP/browser mutations; its error monitor only reports evidence.
A lost browser/control connection or broken device-side prerequisite starts a
new recovery attempt. The browser bundle must implement healthProbe /
healthReport; an older non-empty
TELEOP_WEB_CLIENT_STATIC_DIR/bundle.js is not replaced automatically. OOB
startup checks for the protocol and fails with an asset diagnostic if the local
bundle is too old. For source-tree testing, run
npm run build in deps/cloudxr/webxr_client and point
TELEOP_WEB_CLIENT_STATIC_DIR at its build directory before starting
the service. USB-local static responses use Cache-Control: no-store;
the lifecycle also clears this UI origin’s browser storage once per selected
headset session. Before each local-client CONNECT, CDP reloads the page with
HTTP caching disabled so a previously cached bundle.js cannot mask the
updated client.
The same redacted lifecycle snapshot appears in service status, through
CloudXRLauncher.oob_status(), and at
<cloudxr-install-dir>/run/oob_status.json. A normal shutdown removes the
status file. The selectedSerial and explicitSerial fields identify the
session’s headset selection, and ignoredSerials lists other observed ADB
devices (up to eight display-safe serials). Selection is scoped to the current
service session; a previous status file never pins a new session.
The snapshot exposes separate readiness facts: socketBound means the WSS
listener is serving; transportReady means the selected headset’s network
and, in USB-local mode, TURN and reverse rules have been verified;
clientLoaded means the browser’s CONNECT control is actionable or a
matching browser health report was received; connectDispatched records a
host CONNECT click; and streamConfirmed requires a streaming report with
fresh metrics. readinessStage and readinessSince show the latest
milestone and when it was entered. lastMetricsAt is the host-received
metrics time in epoch milliseconds; the launcher accepts streamConfirmed
only while those metrics and the lifecycle status are less than five seconds
old. Recovery can move the stage backward, and a surviving browser can resume
a stream without another host CONNECT click.
CloudXRLauncher.wait_for_oob_stage("streamConfirmed", timeout_sec=90)
waits for live status and raises on a fatal worker failure or timeout. Call it
after creating the OpenXR session when the application needs a confirmed
headset stream before announcing startup. Starting the standalone CloudXR
service confirms its listener only; headset setup continues with bounded
retries and is visible through the OOB status.
If the OOB lifecycle worker or listening WSS proxy exits unexpectedly, the
service publishes a terminal snapshot with health=fatal and state=FATAL,
clears readiness and streaming flags, and stops the runtime. Unlike a normal
shutdown, it retains oob_status.json so service status can show the
failure reason after the runtime stops. Foreground service run prints one
actionable OOB stopped message and exits nonzero. The per-session WSS
log contains the worker traceback, or a clear message if the worker returned
without an exception. This also applies in hub-only mode. A blocked status
writer does not keep the runtime alive after a fatal worker exit. A writer
stall can delay the status file update; the in-memory fatal snapshot remains
available while the service process is running. An intentional shutdown does
not create a fatal snapshot.
Cadence |
Metric |
Meaning |
|---|---|---|
|
|
Client render rate (FPS), rolling average. |
|
|
Rate at which poses are sent upstream (FPS), rolling average. For teleop this is the rate operator intent reaches the robot. |
|
|
Age of the XR pose at send time. |
|
|
Streamed video rate (FPS), rolling average. |
|
|
Monotonic count of streamed frames. |
|
|
Pose-to-render latency (ms), rolling average. |
|
|
Time spent uploading the pose. |
|
|
Time from pose send to frame received. |
|
|
Share of frames the compositor skipped. |
|
|
Share of frames that arrived out of order. |
|
|
Share of frames whose pose did not match the rendered pixels. |
|
|
Current streaming rate. |
|
|
Estimated available bandwidth. |
|
|
Round-trip time. |
|
|
Packet loss ratio. |
|
|
Average video decode time. |
|
|
Composite network quality, 0-4. |
|
|
Bandwidth component of the quality score, 0-4. |
|
|
Loss component of the quality score, 0-4. |
|
|
Latency component of the quality score, 0-4. |
|
|
Overall session quality, 0-4. Also drives the in-XR quality bars. |
Score metrics use the SDK’s QualityScore scale: 0 NoData, 1 Unsustainable,
2 Degraded, 3 Good, 4 Excellent.
Metrics are cleared when the stream stops, so a new session never reports the previous session’s last known values.
HTTP API#
All endpoints use GET with query parameters on the proxy TLS port.
GET /api/oob/v1/state#
Returns the current hub state: connected headsets, latest metrics, and config version.
curl -k https://localhost:48322/api/oob/v1/state
Example response:
{
"updatedAt": 1712800000000,
"configVersion": 0,
"config": {"serverIP": "10.0.0.1", "port": 48322},
"headsets": [
{
"clientId": "abc-123",
"connected": true,
"deviceLabel": "Quest 3",
"registeredAt": 1712799990000,
"metricsByCadence": {
"frame": {
"at": 1712800000000,
"metrics": {"streaming.framerate": 72.0}
}
}
}
]
}
GET /api/oob/v1/config#
Push config to connected headsets via query parameters:
curl -k "https://localhost:48322/api/oob/v1/config?serverIP=10.0.0.5&port=48322"
Example response:
{
"ok": true,
"changed": true,
"configVersion": 1,
"targetCount": 1
}
Supported query keys: serverIP, port, panelHiddenAtStart, codec.
Optional targetClientId restricts the push to a single headset (returns 404
if not connected).
Authentication#
Set CONTROL_TOKEN=<secret> to require a token on all hub operations.
Pass it as:
WebSocket:
"token"field in theregisterpayloadHTTP:
?token=<secret>query parameter orX-Control-Tokenheader
Web client integration#
The web client connects to the hub when the page URL contains
oobEnable=1 plus serverIP and port:
https://nvidia.github.io/IsaacCapture/client/main/?oobEnable=1&serverIP=10.0.0.1&port=48322
The client builds wss://{serverIP}:{port}/oob/v1/ws and:
Registers as role
"headset"Reports
clientMetricsperiodically (default every 500 ms)Receives
configpushes from operator
URL query parameter overrides#
The following URL parameters override their corresponding form fields (and
localStorage values) so that bookmarked links always take priority over
previously saved settings:
serverIPCloudXR server IP addressportCloudXR server portcodecvideo codecpanelHiddenAtStarthide the control panel on load
When no URL override is present, form fields restore from localStorage, with one
exception: a saved device profile other than Custom is re-applied from the current
profile table at startup, so profile default updates (frame rate, bitrate, codec)
reach returning clients. Only Custom keeps manually edited values.
Environment variables#
Variable |
Description |
|---|---|
|
WSS proxy port (default |
|
Optional auth token for hub access |
|
Override the auto-detected LAN IP in hub initial config |
|
Override the LAN IP used for headset bookmark URLs |
|
Override the web client origin URL |
|
Override the signaling port (default same as proxy port) |
|
Default video codec for headset bookmarks |
|
Hide control panel on load ( |
|
HashRouter fragment appended to the bookmark URL. Default empty
(no fragment — the web client picks its own landing route). Set
to e.g. |
|
Set to any non-empty value (e.g. |
|
Pin the intended headset before starting OOB when more than one ADB
transport is visible. Without it, the first observation must contain
exactly one serial; that serial is pinned even while |
|
Positive finite recovery-episode duration in seconds (default |
|
Positive finite retry and observation interval in seconds (default |
|
Non-negative number of browser-local reconnect attempts (default |
|
Non-negative delay between browser-local attempts in milliseconds
(default |
|
CloudXR backend port the headset reaches via |
|
coturn TURN-server port for WebRTC ICE relay in |
USB-local mode (--setup-oob --usb-local)#
--usb-local routes teleop signalling, the web client, and WebRTC media
over the USB cable on the headset’s loopback via adb reverse. It requires
--setup-oob because the OOB hub is the only path that delivers TURN relay
configuration to the client (without it the headset has no way to discover the
coturn endpoint). Always use both flags together:
python -m isaaccapture.cloudxr.service run --accept-eula --setup-oob --usb-local
On startup the launcher:
Pre-flights host tools, ports, and static assets. It can start with no headset attached.
Resolves the WebXR static directory from
TELEOP_WEB_CLIENT_STATIC_DIR(default~/.cloudxr/static-client) and syncs missingindex.html,bundle.js, andbundle.emulator.jsfrom the published client (see Build WebXR Client).Serves that directory at
https://localhost:48322/client/on the WSS proxy’s existing TLS listener.After the selected headset is ready, checks its non-loopback network, starts/verifies coturn, and creates
adb reverserules for 48322 (WSS and static UI), 49100 (backend), and 3478 (TURN).Launches the teleop URL and clicks CONNECT via CDP. After cable loss, replugging the same serial first rebuilds all three reverse rules, verifies coturn, and repairs CDP forwarding without mutating a surviving browser tab. The WebXR client owns its bounded reconnect attempts. Fresh post-repair stream metrics resume the existing session; a stable terminal event allows one same-tab CONNECT click, and only a missing/unusable tab falls back to full browser launch automation. ADB loss, missing reverse rules, TURN failure or restart, and control/CDP/stream transport loss all enter this preservation-first repair path. While preserving an active browser, the launcher checks recovery prerequisites at most once per second instead of waiting for the ordinary five-second observation cadence.
In --usb-local mode the launcher also wipes localStorage / IndexedDB /
cookies / HTTP cache for the teleop UI origin (https://localhost:<proxy_port>)
once for the selected session — the SDK and web client both cache settings
(e.g. general.iceTransportPolicy for ICE transport policy,
cxr.isaac.teleopPath for the last-used project) in localStorage, and a
stale value can silently win over a fresh URL param. The origin is owned
by the launcher so clearing it has no collateral effect; the step is a
no-op when the headset browser isn’t running yet.
Required apt packages: adb (android-tools-adb) and coturn.
No Node.js / npm is required at runtime.
Troubleshooting#
Teleop client error: “No local connection candidates” (0xC0F2220F)#
Cause: Wi-Fi must stay associated on the headset throughout the
session, even in --usb-local mode. No teleop traffic actually flows
over Wi-Fi — every byte goes over the USB cable via adb reverse —
but Chromium’s WebRTC rtc::NetworkManager excludes loopback
interfaces when enumerating networks for ICE. If the only active
network on the headset is lo, ICE gathering hangs at gathering
forever (no local candidates emitted, no error fires) until the CloudXR
session times out with this code.
Fix: Associate the headset with any Wi-Fi network and retry. Internet is not required — a phone hotspot with no SIM works, an open AP you never authenticate to works. The packets still route over USB (the kernel short-circuits loopback regardless of source interface); the Wi-Fi interface just needs to exist with an IP so WebRTC’s enumeration is non-empty.
The lifecycle checks this via adb shell ip -o -4 addr show after the
host is listening. A successful command with no non-loopback interface
reports a network block; a failed ADB command reports cable/transport loss
and never claims that Wi-Fi dropped. Wi-Fi restoration resumes recovery.
Web client UI has no text (OOB / --host-client)#
Cause: Production webpack emits bundle.js (main app, including UIKit MSDF text)
and bundle.emulator.js (desktop emulation only). If the static cache under
TELEOP_WEB_CLIENT_STATIC_DIR is missing or outdated — especially a truncated
bundle.js — in-VR text does not render.
Fix: Ensure index.html, bundle.js, and bundle.emulator.js are present
under the static dir — run the launcher once with network, copy a full npm run build
output, or use the GitHub Pages URL directly. See
Build WebXR Client.
coturn not found / TURN server failed to start#
Cause: --usb-local requires coturn to relay WebRTC media between
the headset (TCP via adb reverse) and the CloudXR backend (UDP on
loopback).
Fix: sudo apt-get install -y coturn. The launcher starts its own
turnserver process on 127.0.0.1:3478; no systemd service is needed
(and may conflict — stop the system coturn.service if enabled).
Inspect /tmp/coturn-cloudxr-3478.log for bind errors or credential
mismatches; the launcher truncates this file on every start so only the
current session’s output is present.
Tab not found within timeout#
Cause: The headset’s default URL handler is something other than Meta Quest Browser (e.g. WebLayer on Meta Quest) and did not open the teleop URL in a browser with remote-debugging exposed.
Fix: Open chrome://inspect#devices on this PC, inspect the
headset tab manually, and click CONNECT. Or set a different default
browser on the headset.
WebXR static download fails (offline / proxy)#
Cause: The launcher syncs index.html, bundle.js, and
bundle.emulator.js into the static dir on first run. Behind a proxy or with no
internet, this fails and --usb-local aborts.
Fix: Pre-stage the full client build/ tree (or the published
/client/<version>/ directory) into the static dir, then re-run. The
launcher only downloads missing or empty files. Override the target directory
via TELEOP_WEB_CLIENT_STATIC_DIR.
Fix: Set the SDK versions in deps/cloudxr/.env (copy from
.env.default) so the download script can resolve the right version,
or stage nvidia-cloudxr-<version>.tgz in deps/cloudxr/ manually.