Camera Streaming#

camera_viz is the reference camera-streaming sample built on Televiz (isaaccapture.viz). It captures frames from one or more cameras and streams them to an XR headset — one plane per camera, aspect-fit — either directly or from a robot to a workstation over the network (split mode). It can also stream recorded camera data — a video file replayed in place of a live camera — and render to a desktop window instead of the headset.

The sample lives at examples/camera_viz/. This page walks you from setup and a camera-free first run to real cameras and the robot → workstation split mode. For the exact command surface and flags, see the README.

camera_viz in window mode with synthetic multi-camera feeds — `camera_viz` in XR mode — one plane per camera.#

Requirements #

A workstation meeting the system requirements (Ubuntu, NVIDIA GPU, CUDA driver) — every source hands frames to the renderer GPU-resident via CuPy.
For Jetson platforms:
- Orin: JetPack 6.2.x or 7.2.1
- Thor: JetPack 7.x
For the default XR mode, a headset to connect as the CloudXR client — follow the quick start step 5. Connect an XR headset. The viewer launches the CloudXR runtime itself; nothing to start separately. No headset handy? --mode window renders to a desktop window instead.
CUDA development components: matching runtime, NVRTC, and headers for CuPy kernels. JetPack 7.2.1 uses CUDA 13.2; see Jetson CUDA prerequisites.
Camera — optional, recommended: USB / UVC, OAK-D, or supported ZED camera for a live feed. A recording or synthetic pattern works without a camera; ZED SDK capture needs setup --with-zed.
XR controllers — optional: for adjusting the view with Controller bindings.
CloudXR setup: see Quick Start for first-launch requirements and firewall configuration.

Setup #

Clone the repository if you haven’t already (quick start step 1. Check out code base (for examples)), then run the sample’s one-time setup:

examples/camera_viz/camera_viz.sh setup
source examples/camera_viz/.venv/bin/activate

On Jetson, add --jetson. If CUDA components are missing, see Jetson CUDA prerequisites below.

There is no need to install the isaaccapture pip package yourself. setup builds the sample’s own environment: isaaccapture[cloudxr] — the cloudxr extra is not optional here, since XR is the default display mode and the viewer launches the runtime itself — plus every other Python dependency, into .venv/ via uv. It resolves a release matching the checkout’s major/minor version, falling back to a release candidate and then, after asking, to a source build of the surrounding checkout (see scripts/_install_deps.sh; --wheel and --build-from-source override the choice). When required system packages are missing, it prints the exact apt-get line to approve — declining, or a non-interactive run, aborts.

Setup selects CuPy using CUDA_PATH, the toolkit containing nvcc on PATH, or /usr/local/cuda, in that order. It checks the loaded NVRTC version and executes a GPU kernel before reporting success.

By default setup provisions the direct-mode path — USB / UVC and OAK-D camera support. Split mode (RTP) and ZED support are opt-in, since both pull in dependencies the direct path never needs. Flags trim or extend that:

Flag	Effect
`--no-v4l2`	Skip USB / UVC webcam support (`opencv-python`).
`--no-oakd`	Skip OAK-D support (`depthai`).
`--with-rtp`	Also install the split-mode dependencies: the GStreamer system packages, PyGObject, and the native NVENC/NVDEC codec build. Required for `loopback` and for any config with `source: rtp`; implied by `--sender-only`. Direct mode does not need it.
`--with-zed`	Also build + install the ZED SDK’s Python API (`pyzed`). Requires the ZED SDK on the machine (default `/usr/local/zed`; override with `--zed-sdk PATH`).
`--sender-only`	Split mode only — robot-side install of just the sender’s dependencies.
`--jetson`	Enable Jetson CUDA library discovery for direct or split mode. With RTP enabled, also check for the matching NVRTC package.
`--venv PATH`	Use `PATH` for the environment and link the sample’s `.venv` to it; see below.
`--wheel PATH`	Install a locally built `isaaccapture` wheel instead of resolving one from the index — for developing Isaac Capture itself.
`--build-from-source`	Skip the package index and build `isaaccapture` from the surrounding checkout without prompting — a full C++ / CUDA / Vulkan build (see Build from Source).

Using an external virtual environment #

To reuse a Python 3.12 environment, run from the repository root:

examples/camera_viz/camera_viz.sh setup --venv /absolute/path/to/venv

Setup installs dependencies there and links examples/camera_viz/.venv to it. Move an existing .venv directory aside first. The wrapper uses .venv/bin/python regardless of the active shell environment; activation is optional. Add --jetson on Jetson.

Jetson CUDA prerequisites #

CuPy compiles GPU kernels at runtime using NVRTC and CUDA headers (CuPy requirements). If these components are already installed, skip this step.

On JetPack 7.2.1 / CUDA 13.2, the JetPack CUDA development package provides them. With the matching NVIDIA repositories configured:

sudo apt update
sudo apt install nvidia-cuda-dev

Use the CUDA toolkit matching your JetPack. --jetson configures library discovery. A separate nvcc compiler is needed for CUDA source builds, such as the native RTP codec, rather than ordinary CuPy kernel execution.

First run — a recording, no camera required #

The recorded-camera source (type: video) replays a video file through exactly the same source → layer → Televiz path a live camera uses. It is the supported way to stream recorded camera data, it needs no extra setup, and it is the quickest end-to-end check. A test clip ships with the repo, and configs/replay.yaml already points at it. Fetch the clip with Git LFS if your checkout contains only its pointer:

git lfs pull --include="examples/camera_viz/test_data/recording.mp4"

Keep OpenCV installed for replay (--no-v4l2 skips it). From the repository root:

cd examples/camera_viz
./camera_viz.sh run configs/replay.yaml                 # XR headset (default)
./camera_viz.sh run configs/replay.yaml --mode window   # desktop window instead

In XR mode the viewer first brings up the CloudXR runtime (accept the EULA on first launch, or pass --accept-eula). Open the printed client URL in your headset browser, accept the host’s certificate, and click Connect, following 5. Connect an XR headset. On Jetson Orin, select H.264 in the client’s Video Codec setting before connecting.

You should see the terminal report the session and the source coming up:

camera_viz: source=local, mode=xr, xr=True, shapes=quad, 1 layer(s)
[video] opening...
[video] connected
[video] streaming

and the clip looping on a plane in the headset once it connects — or in a desktop window (mode=window, xr=False) with the --mode window override, which starts no runtime.

First-session checklist #

With the default replay.yaml:

Playback: one plane is visible and the clip loops.
Head movement: small turns and leaning change your viewpoint; the plane stays anchored.
Lazy recentering: look more than 45° away for about a second; the plane moves smoothly back in front of you.
Image quality: no freezing, flicker, torn regions, or unexpected colors while still or moving. This mono clip should show the same source image in both eyes.
Controllers, if available: right-stick click recenters; A changes lock mode; Y restores the config.
Restart: press Ctrl+C, rerun, and reconnect if needed; playback resumes.

In window mode, check playback, image quality, and restart. If a check fails, see Troubleshooting.

Replaying your own recording #

Point path at any file OpenCV’s FFmpeg backend reads — mp4 / mkv / webm carrying H.264, HEVC, AV1, and so on. The file is probed once at startup for its size and frame rate (a missing or absurd rate falls back to 30 fps); a missing or unreadable file is a configuration error and fails immediately instead of retrying like a camera would. A complete config:

source: local

cameras:
  - name: replay
    enabled: true
    type: video
    path: ../test_data/recording.mp4   # required; relative to this YAML's directory
    loop: true                         # false = hold the last frame at end of file
    fps: 0                             # 0 = the file's native rate
    stereo: false                      # true = split a side-by-side recording into eyes
    # width/height default to the file's native size (per eye when stereo)

display:
  mode: xr                             # or: window
  placements:
    replay: { lock_mode: lazy, distance: 1.5 }

With source: rtp, width, height, and fps become required — the sender paces its encode loop at fps and the receiver sizes its decoder from the config — and stereo is rejected, since the sender needs per-eye streams. A mono recording is otherwise a fine capture side for split mode and for loopback, which exercises the whole encode → UDP → decode path with no camera attached.

Supported sources #

The source kind is selected by the type field of each entry in the YAML cameras list:

`type:`	Notes
`v4l2`	USB / UVC cameras — anything `v4l2-ctl --list-formats-ext` reports.
`oakd`	OAK-D RGB / LEFT / RIGHT; mono or `stereo: true`. Stereo ships GRAY8 over USB to halve bandwidth and is broadcast to RGBA on the GPU; `stereo_rgb` keeps color.
`zed`	ZED 2 / Mini / X One; mono or `stereo: true` (per-eye SDK retrieve, zero-copy on the GPU).
`video`	Recorded camera data — video-file replay (anything OpenCV’s FFmpeg backend reads). Loops by default; `stereo: true` splits side-by-side recordings into eyes (viewer only). See First run — a recording, no camera required.
`synthetic`	Debugging tool — GPU-generated test pattern, no camera or file required.

Running with a real camera #

Attach the camera to the machine that runs the viewer, keep source: local in the config, and run with the matching config. First edit its device, resolution, and frame rate to match your camera. In particular, v4l2.yaml defaults to a ZED Mini’s 2560×720 UVC mode; a regular webcam needs values it supports (inspect them with v4l2-ctl --list-formats-ext):

./camera_viz.sh run configs/v4l2.yaml     # or oakd.yaml / zed.yaml

You should see the same startup lines as above with the camera’s tag ([v4l2], [oakd], [zed]) and the live feed. Multiple entries in the cameras list render as one plane each.

Display modes #

XR is the default: each camera renders as its own plane in the headset via the active OpenXR runtime, and stereo sources (stereo: true) render true side-by-side stereo. Pass --mode window (or set display.mode: window in the YAML) to render to a desktop window instead — no headset or runtime needed; stereo shows the left eye.

In XR, how a plane follows the operator’s head is the per-camera lock_mode under display.placements.<name>:

Mode	Behavior
`world`	Placed once in front of you and stays put.
`head`	Follows your head every frame. Head-locked content updates its pose at application rate, so it trails fast head motion by roughly a frame — expected for any head-locked OpenXR layer; prefer `lazy` unless you need a true HUD.
`gimbal`	Follows your position but not your rotation: the surface stays pointed where you first looked, walks with you, and turning your head looks around it. The natural mode for wide cylinder feeds (a “virtual gimbal”).
`lazy`	World-locked, but re-snaps in front of you when you look away (default).

Lazy-mode knobs live under placements.<name>: look_away_angle_deg, reposition_distance, reposition_delay_s, transition_duration_s.

Controller bindings #

In XR the controllers retune the view live, without editing the YAML and restarting. The right hand changes how the feed looks; the left, what surface it is mapped onto:

Input	Effect
Right stick ←/→	Stereo plane gap — how far apart the two eyes’ planes sit (`placements.<cam>.stereo_plane_distance_cm`). Widening it pushes the scene back instead of packing it into the space in front of the planes. Stereo cameras only; on `equirect`, which has no gap to set, the same stick pans the panorama.
Right stick click	Recenter on your view: an `equirect` yaws so the middle of the panorama lands dead ahead, and a placed surface re-snaps its anchor.
`A`	Cycle the lock mode: `world` → `head` → `gimbal` → `lazy`.
`B`	Toggle mono / stereo. Stereo cameras only.
`X`	Cycle the shape: `quad` → `cylinder` → `equirect`.
`Y`	Reset everything to the YAML values.
Left stick	Retunes the active shape: `quad` size / vertical position, `cylinder` arc width / vertical position, `equirect` horizontal / vertical span.

Changes apply to every camera at once and appear both on the terminal status panel and on a head-locked panel in the headset that auto-hides shortly after. Neither toggle reallocates: B sends the left frame to both eyes, and X flips visibility between shapes built at startup, so the extra shapes cost VRAM (reported at startup) rather than a stall on the press.

The stereo gap is bounded below divergent parallax — a gap wider than your IPD would need the eyes to splay outward — and the headset’s measured IPD sets that ceiling, not the config. The HUD suggests a gap derived from the plane distance and that IPD.

Bindings, rates and limits live under display.controls; see the README for the full set.

Display surfaces #

By default each camera renders on a flat plane, which suits normal-FOV feeds. Wide-FOV and panoramic sources look better on a curved surface: set shape per camera under display.placements.<name>:

Shape	Behavior
`quad` (default)	Flat plane. All lock modes apply.
`cylinder`	Curved arc facing the operator, so the image stays at a constant viewing distance edge to edge. `cylinder_radius_m` (default 2.0) sets that distance, `cylinder_angle_deg` (default 90) the arc width. All lock modes apply (`head` = gaze-tracking curved visor).
`equirect`	Full 360°×180° sphere around the operator, for equirectangular panorama / VR-video sources. The middle of the texture points along `equirect_yaw_deg` (0 = the direction the headset last recentered on, positive to the left), which is how a camera mounted facing another way gets aimed. Lock modes don’t apply — the sphere is centred on the operator and follows them everywhere.

Curved shapes exist only in XR mode — the viewer exits with an error in window mode. Stereo sources render per-eye textures on the same surface, and stereo_baseline_mm adds a per-eye pose shift (no effect on the equirect sphere at its default infinite radius).

Surfaces are composited by the OpenXR runtime, which keeps them sharp under head motion and lets CloudXR stream them efficiently (see OpenXR composition layers). For flat planes only, compositor: televiz opts a camera back into Televiz’s built-in compositor. On Jetson Orin (CloudXR experimental runtime), set compositor: televiz so the plane is not left black; see Troubleshooting below. On the Televiz 2D API the same switch is openxr_composition = False (see Jetson Orin, XR).

CloudXR runtime flags #

In XR mode the viewer attaches to the running CloudXR runtime, or starts one if none is serving. Useful flags:

--accept-eula — accept the CloudXR EULA non-interactively (first run only).
--cloudxr-device-profile PROFILE — NV_DEVICE_PROFILE (default Quest3).
--no-host-client — disable serving CloudXR.js at https://<host>:48322/client/. Only applies when this run starts the service; stop and restart the service to change hosting.

When the viewer starts a CloudXR service with hosting enabled (the default; --no-host-client disables it), it prints the local /client/ URL. To inspect or stop an already-running service, activate the example venv (or any environment with isaaccapture) and run:

python -m isaaccapture.cloudxr.service status
python -m isaaccapture.cloudxr.service stop

Run camera_viz.py --help for the rest (install dir, env-config file, WSS proxy toggle).

Runtime settings themselves are declared in the config rather than exported. The viewer ships defaults (pose wait off, runtime foveation on), and display.cloudxr overrides them per deployment — list only what you want to change:

display:
  cloudxr:
    NV_DEVICE_PROFILE: apple-vision-pro
    NV_ENABLE_POSE_WAIT: null   # null drops a viewer default

The viewer writes these to a generated --cloudxr-env-config file, which outranks the process environment — so a value left in your shell (for example after sourcing ~/.cloudxr/run/cloudxr.env, as --no-launch-cloudxr-runtime suggests) cannot silently override the config, and neither can it override --cloudxr-device-profile. Booleans are written in the lowercase spelling the runtime’s parser recognises; the launcher-computed keys (XR_RUNTIME_JSON, NV_CXR_RUNTIME_DIR, NV_CXR_OUTPUT_DIR, XRT_NO_STDIN) are rejected by name. Passing --cloudxr-env-config yourself takes precedence over the generated file.

An unknown variable name warns with a suggestion before the runtime launches, and is passed through regardless — the runtime has more settings than the viewer lists. To confirm what the runtime resolved, read the settings dump at the top of the newest ~/.cloudxr/logs/cxr_server.*.log.

Split mode — robot → workstation over RTP #

Split mode runs the capture side on the robot (camera_streamer.py) and ships RTP H.264 to the viewer on the workstation (source: rtp).

Warning

Split mode is not recommended in most cases. It exists for one situation: the cameras are on the robot, but Isaac Teleop runs on a workstation, so the frames must be streamed to where Isaac Teleop is running. That costs a full extra encode/decode hop — NVENC on the robot, UDP, NVDEC on the workstation — so whenever a camera can attach directly to the machine running Isaac Teleop, run direct mode instead. Wired networks only: there is no retransmit or FEC — one lost packet corrupts one frame until the next IDR (default every 5 s).

In split mode every camera entry must pin width, height, and fps in the YAML — the receiver sizes its decoder from the config, not from the wire.

The workstation needs the RTP dependencies, which setup does not install by default — re-run it with --with-rtp if you provisioned for direct mode first. The robot side is handled by deploy, which implies the flag.

Set source: rtp in the config, export the robot/streaming credentials once per shell, then deploy the sender and run the viewer:

export REMOTE_HOST=10.0.0.5 REMOTE_USER=nvidia
export STREAMING_HOST=10.0.0.42                  # workstation IP

./camera_viz.sh deploy configs/v4l2.yaml         # full deploy + systemd unit on the robot
./camera_viz.sh run    configs/v4l2.yaml         # viewer on the workstation

deploy rsyncs the source to the robot, installs sender dependencies, renders a camera-streamer.service systemd user unit (injecting --host from $STREAMING_HOST without editing the YAML on disk), and enables it. Operate the running unit with ./camera_viz.sh service-{status,logs,restart}. The sender retries forever across unplug, SDK errors, and network blips.

Loopback #

Loopback is a testing / debugging aid, not a deployment mode: ./camera_viz.sh loopback configs/v4l2.yaml runs the sender and viewer together on 127.0.0.1 — the quickest way to smoke-test the RTP path on one machine. It also works camera-free with a mono type: video entry: ./camera_viz.sh loopback configs/replay.yaml runs as shipped.

Two things to know when running it:

Use the config that matches the attached camera. configs/v4l2.yaml is a ZED Mini template (/dev/video0, 2560x720); on any other camera it fails to open and the viewer reports submit/s 0.0. For a RealSense, run configs/realsense.yaml.
loopback does not forward --mode, so it opens in the YAML’s display.mode (xr by default). Set display.mode: window in the YAML for a desktop window.

Configuration #

A single YAML drives both capture and visualization. Each entry in the cameras list becomes its own plane (and, in split mode, its own RTP port). Abbreviated:

source: local | rtp
streaming:
  host: 192.168.1.100         # workstation IP (overridden at deploy time)
encoder: auto | native | gstreamer

cameras:
  - name: cam
    enabled: true
    type: v4l2                # v4l2 | oakd | zed | video | synthetic
    width: 2560               # video: optional — defaults to the file's size
    height: 720               # (required when source: rtp)
    fps: 30
    stereo: false             # zed / video / synthetic — per-eye capture + SBS in XR
    path: clip.mp4            # video only — file to replay, relative to this YAML
    loop: true                # video only — rewind at end of file
    rtp:
      port: 5000              # left eye when stereo
      port_right: 5001        # required when stereo + source: rtp
      bitrate_mbps: 15

display:
  mode: xr | window           # default: xr
  window: { width, height }
  xr:
    near_z: 0.05              # default
    far_z: 100.0              # default
    system_wait_seconds: -1   # default; wait forever (0 fails fast)
  clear_color: [r, g, b, a]
  placements:
    cam:
      lock_mode: lazy         # world | head | lazy | gimbal
      distance: 1.5
      # size: [w_m, h_m]
      # stereo_plane_distance_cm: 0   # gap between the eyes' planes
      # shape: quad           # quad | cylinder | equirect (cylinder/equirect are XR-only)
      # equirect_yaw_deg: 0.0  # equirect: heading the middle of the feed points at
      # compositor: openxr    # openxr (default) | televiz — quads only
      # cylinder_radius_m: 2.0
      # cylinder_angle_deg: 90

See the configs/ directory for a complete, commented YAML per source kind.

Troubleshooting #

The XR session fails to create — check ~/.cloudxr/logs/cxr_server.*.log and ~/.cloudxr/logs/runtime_worker_stderr.log (runtime_stderr.log with ISAACCAPTURE_LOGGING=off) for the startup failure. XR_ERROR_FORM_FACTOR_UNAVAILABLE (-35) means the runtime is up but no headset has clicked CONNECT yet; XR mode waits according to display.xr.system_wait_seconds (default -1 to wait indefinitely; 0 fails fast; positive values wait that many seconds; override with --xr-wait). Pass --mode window to render to a desktop window instead (no runtime involved).
No window appears over SSH — --mode window needs a local display; run on the machine you’re sitting at, or use a video-capable remote desktop.
“video source: no such file” — relative path: values resolve against the YAML’s directory (configs/), not the directory you launched from.
A source fails asking for CuPy / CUDA — check the matching CUDA headers, NVRTC, and /usr/local/cuda target (Jetson CUDA prerequisites on Jetson), then rerun setup. nvidia-smi checks driver support, not installed headers. To test GPU execution, run from examples/camera_viz and expect [1 2 3 4]:
```
.venv/bin/python -c 'import cupy as cp; print((cp.arange(4) + 1).get())'
```
Python cannot find dependencies — use the sample’s .venv/bin/python, or select an environment with setup –venv.
No video on Orin. When the CloudXR runtime runs on Jetson Orin, set Video Codec to H.264 in the CloudXR web client. See 5. Connect an XR headset.
Black camera plane on Orin — with the CloudXR experimental runtime on Orin, the default OpenXR compositor shows the camera plane but leaves its contents black. Under the camera’s display.placements entry, set compositor: televiz (quads only). The same feed displays correctly with this workaround:
```
display:
  placements:
    cam:
      compositor: televiz
```
Split mode renders nothing — check the sender is up (./camera_viz.sh service-status), $STREAMING_HOST was the workstation’s IP at deploy time, and UDP ports (default 5000+) aren’t firewalled.
CUDA kernel errors — rerun the setup check from examples/camera_viz. It prints CUDA diagnostics on failure; a driver version from nvidia-smi does not identify the installed toolkit:
```
.venv/bin/python scripts/check_cuda.py
```
Thor: ``CUDA_ERROR_NO_BINARY_FOR_GPU`` — an older NVRTC compiler can produce kernels incompatible with Thor. Select your JetPack’s CUDA 13.x toolkit and rerun setup. As a temporary workaround for CuPy kernels, try:
```
CUPY_COMPILE_WITH_PTX=1 ./camera_viz.sh run configs/synthetic.yaml
```
This lets the driver compile the kernel for the GPU (CuPy PTX option).
Not sure which side is stuck? — set verbose: true at the top of the YAML for periodic per-source breadcrumbs on both ends.

How it works #

The sample is organized so that capture, transport, and visualization are cleanly separated, with Televiz as the compositor at the end of the chain:

camera_viz/
├── camera_viz.sh        — CLI: setup / loopback / run / deploy / service-*
├── camera_viz.py        — receiver / viewer (drives a Televiz VizSession)
├── camera_streamer.py   — robot-side RTP sender (per-camera supervisor)
├── pipeline/            — source ABC + threaded runner
├── placements/          — XR lock-mode strategies (world / head / lazy / gimbal)
├── sources/             — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264
├── transports/          — RTP sender + receiver (native + GStreamer)
├── codec/               — native NVENC / NVDEC pybind module
├── configs/             — one YAML per source kind
├── test_data/           — sample replay clip (Git LFS)
└── scripts/             — installer + systemd unit template

Sources (sources/) implement a common source ABC and hand frames to a threaded runner in pipeline/. Each source produces GPU frames where possible — e.g. the ZED source uses retrieve_image(MEM.GPU) so BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip.
The viewer (camera_viz.py) creates a VizSession and adds one QuadLayer per enabled camera, then submits each frame to its layer and calls render() once per frame. Stereo cameras submit both eyes.
Transport (transports/) carries the split mode: an RTP H.264 sender on the robot and a receiver on the workstation, with a native NVENC/NVDEC codec module (codec/) or a GStreamer fallback.
Placement (placements/) holds the XR lock-mode strategies. Placement is application policy — Televiz only renders a layer at whatever pose the app sets each frame.

Next steps #

Connect a live camera or replay your own recording.
Adjust the view and save placement settings in YAML.
Stream from a robot when capture and viewing run on separate machines.
Share the XR session with teleoperation and map inputs with the retargeting interface.