Camera Streaming#

camera_viz is the reference camera-streaming sample built on Televiz (isaacteleop.viz). It captures frames from one or more cameras and streams them to an XR headset — one plane per camera, aspect-fit — either directly or from a robot to a workstation over the network (split mode). For development and debugging it can also replay a video file in place of a camera, and render to a desktop window instead of the headset.

The sample lives at examples/camera_viz/. This page walks you from setup and a hardware-free first run to real cameras and the robot → workstation split mode. For the exact command surface and flags, see the README.

camera_viz in window mode with synthetic multi-camera feeds — `camera_viz` in XR mode — one plane per camera.#

Requirements #

A workstation meeting the system requirements (Ubuntu, NVIDIA GPU, CUDA driver) — every source hands frames to the renderer GPU-resident via CuPy.
For Jetson platforms, use JetPack 6.2.1 on Orin or JetPack 7.1 on Thor.
For the default XR mode, a headset to connect as the CloudXR client — follow the quick start step 5. Connect an XR headset. The viewer launches the CloudXR runtime itself; nothing to start separately. No headset handy? --mode window renders to a desktop window instead.

Setup #

Clone the repository if you haven’t already (quick start step 1. Check out code base (for examples)), then run the sample’s one-time setup:

examples/camera_viz/camera_viz.sh setup
source examples/camera_viz/.venv/bin/activate

There is no need to install the isaacteleop pip package yourself — setup creates the sample’s own environment: it installs isaacteleop>=1.4 (which bundles Televiz) and every other Python dependency from PyPI into .venv/ via uv, builds the native NVENC/NVDEC codec, and probes system packages (GStreamer plugins, cairo / girepository headers, JetPack cuda-nvrtc + ld.so wiring). When something is missing it prints the exact apt-get line and prompts [y/N] — answering n or running non-interactively aborts.

By default setup provisions everything except ZED support; flags trim or extend that:

Flag	Effect
`--no-v4l2`	Skip USB / UVC webcam support (`opencv-python`).
`--no-oakd`	Skip OAK-D support (`depthai`).
`--no-rtp`	Skip split-mode dependencies: the GStreamer system packages and the native NVENC/NVDEC codec build. Direct mode still works.
`--with-zed`	Also build + install the ZED SDK’s Python API (`pyzed`). Requires the ZED SDK on the machine (default `/usr/local/zed`; override with `--zed-sdk PATH`).
`--sender-only`	Split mode only — robot-side install of just the sender’s dependencies.
`--jetson`	Split mode only — extra CUDA wiring JetPack images need on the robot.
`--venv PATH`	Install into an existing virtual environment instead of creating `.venv/`.
`--wheel PATH`	Install a locally built `isaacteleop` wheel instead of the PyPI release — for developing Isaac Teleop itself (see Build from Source).

First run — no camera required #

The video-replay source (type: video) plays a recording through exactly the same path a live camera uses, so it doubles as the quickest end-to-end check and as a stand-in feed while the real camera isn’t available. A test clip ships with the repo and configs/replay.yaml already points at it:

cd examples/camera_viz
./camera_viz.sh run configs/replay.yaml                 # XR headset (default)
./camera_viz.sh run configs/replay.yaml --mode window   # desktop window instead

In XR mode the viewer first brings up the CloudXR runtime (accept the EULA on first launch, or pass --accept-eula), then you should see the terminal report the session and the source coming up:

camera_viz: source=local, mode=xr, xr=True, shapes=quad, 1 layer(s)
[video] opening...
[video] connected
[video] streaming

and the clip looping on a plane in the headset once it connects — or in a desktop window (mode=window, xr=False) with the --mode window override, which starts no runtime.

To replay your own video, set a custom path: in configs/replay.yaml — relative paths resolve against the YAML’s directory. loop: false holds the last frame instead of rewinding; stereo: true replays a side-by-side recording (e.g. from a ZED) as stereo, splitting each frame into per-eye views (direct mode only).

Supported sources #

The source kind is selected by the type field of each entry in the YAML cameras list:

`type:`	Notes
`v4l2`	USB / UVC cameras — anything `v4l2-ctl --list-formats-ext` reports.
`oakd`	OAK-D mono RGB / LEFT / RIGHT (stereo not yet wired).
`zed`	ZED 2 / Mini / X One; mono or `stereo: true` (per-eye SDK retrieve, zero-copy on the GPU).
`video`	Video-file replay (anything OpenCV’s FFmpeg backend reads). Loops by default; `stereo: true` splits side-by-side recordings into eyes (viewer only).
`synthetic`	Debugging tool — GPU-generated test pattern, no hardware or file.

Running with a real camera #

Attach the camera to the machine that runs the viewer, keep source: local in the config, and run with the matching config:

./camera_viz.sh run configs/v4l2.yaml     # or oakd.yaml / zed.yaml

You should see the same startup lines as above with the camera’s tag ([v4l2], [oakd], [zed]) and the live feed. Multiple entries in the cameras list render as one plane each.

Display modes #

XR is the default: each camera renders as its own plane in the headset via the active OpenXR runtime, and stereo sources (stereo: true) render true side-by-side stereo. Pass --mode window (or set display.mode: window in the YAML) to render to a desktop window instead — no headset or runtime needed; stereo shows the left eye.

In XR, how a plane follows the operator’s head is the per-camera lock_mode under display.placements.<name>:

Mode	Behavior
`world`	Placed once in front of you and stays put.
`head`	Follows your head every frame. Head-locked content updates its pose at application rate, so it trails fast head motion by roughly a frame — expected for any head-locked OpenXR layer; prefer `lazy` unless you need a true HUD.
`gimbal`	Follows your position but not your rotation: the surface stays pointed where you first looked, walks with you, and turning your head looks around it. The natural mode for wide cylinder feeds (a “virtual gimbal”).
`lazy`	World-locked, but re-snaps in front of you when you look away (default).

Lazy-mode knobs live under placements.<name>: look_away_angle_deg, reposition_distance, reposition_delay_s, transition_duration_s.

Display surfaces #

By default each camera renders on a flat plane, which suits normal-FOV feeds. Wide-FOV and panoramic sources look better on a curved surface: set shape per camera under display.placements.<name>:

Shape	Behavior
`quad` (default)	Flat plane. All lock modes apply.
`cylinder`	Curved arc facing the operator, so the image stays at a constant viewing distance edge to edge. `cylinder_radius_m` (default 2.0) sets that distance, `cylinder_angle_deg` (default 90) the arc width. All lock modes apply (`head` = gaze-tracking curved visor).
`equirect`	Full 360°×180° sphere around the operator, for equirectangular panorama / VR-video sources. Lock modes don’t apply.

Curved shapes exist only in XR mode — the viewer exits with an error in window mode. Stereo sources render per-eye textures on the same surface, and stereo_baseline_mm adds a per-eye pose shift (no effect on the equirect sphere at its default infinite radius).

Surfaces are composited by the OpenXR runtime, which keeps them sharp under head motion and lets CloudXR stream them efficiently (see OpenXR composition layers). For flat planes only, compositor: televiz opts a camera back into Televiz’s built-in compositor.

CloudXR runtime flags #

In XR mode the viewer launches the CloudXR runtime and WSS proxy itself. Useful flags:

--no-launch-cloudxr-runtime — reuse an already-running runtime.
--accept-eula — accept the CloudXR EULA non-interactively (first run only).
--cloudxr-device-profile PROFILE — NV_DEVICE_PROFILE (default Quest3).

Run camera_viz.py --help for the rest (install dir, env-config file, WSS proxy toggle).

Split mode — robot → workstation over RTP #

Split mode runs the capture side on the robot (camera_streamer.py) and ships RTP H.264 to the viewer on the workstation (source: rtp).

Warning

Split mode is not recommended in most cases. It exists for one situation: the cameras are on the robot, but Isaac Teleop runs on a workstation, so the frames must be streamed to where Isaac Teleop is running. That costs a full extra encode/decode hop — NVENC on the robot, UDP, NVDEC on the workstation — so whenever a camera can attach directly to the machine running Isaac Teleop, run direct mode instead. Wired networks only: there is no retransmit or FEC — one lost packet corrupts one frame until the next IDR (default every 5 s).

In split mode every camera entry must pin width, height, and fps in the YAML — the receiver sizes its decoder from the config, not from the wire.

Set source: rtp in the config, export the robot/streaming credentials once per shell, then deploy the sender and run the viewer:

export REMOTE_HOST=10.0.0.5 REMOTE_USER=nvidia
export STREAMING_HOST=10.0.0.42                  # workstation IP

./camera_viz.sh deploy configs/v4l2.yaml         # full deploy + systemd unit on the robot
./camera_viz.sh run    configs/v4l2.yaml         # viewer on the workstation

deploy rsyncs the source to the robot, installs sender dependencies, renders a camera-streamer.service systemd user unit (injecting --host from $STREAMING_HOST without editing the YAML on disk), and enables it. Operate the running unit with ./camera_viz.sh service-{status,logs,restart}. The sender retries forever across unplug, SDK errors, and network blips.

Loopback #

Loopback is a testing / debugging aid, not a deployment mode: ./camera_viz.sh loopback configs/v4l2.yaml runs the sender and viewer together on 127.0.0.1 — the quickest way to smoke-test the RTP path on one machine. It also works camera-free with a mono type: video entry (set width / height / fps).

Configuration #

A single YAML drives both capture and visualization. Each entry in the cameras list becomes its own plane (and, in split mode, its own RTP port). Abbreviated:

source: local | rtp
streaming:
  host: 192.168.1.100         # workstation IP (overridden at deploy time)
encoder: auto | native | gstreamer

cameras:
  - name: cam
    enabled: true
    type: v4l2                # v4l2 | oakd | zed | video | synthetic
    width: 2560               # video: optional — defaults to the file's size
    height: 720               # (required when source: rtp)
    fps: 30
    stereo: false             # zed / video / synthetic — per-eye capture + SBS in XR
    path: clip.mp4            # video only — file to replay, relative to this YAML
    loop: true                # video only — rewind at end of file
    rtp:
      port: 5000              # left eye when stereo
      port_right: 5001        # required when stereo + source: rtp
      bitrate_mbps: 15

display:
  mode: xr | window           # default: xr
  window: { width, height }
  xr:     { near_z, far_z }
  clear_color: [r, g, b, a]
  placements:
    cam:
      lock_mode: lazy         # world | head | lazy | gimbal
      distance: 1.5
      # size: [w_m, h_m]
      # stereo_baseline_mm: 0
      # shape: quad           # quad | cylinder | equirect (cylinder/equirect are XR-only)
      # compositor: openxr    # openxr (default) | televiz — quads only
      # cylinder_radius_m: 2.0
      # cylinder_angle_deg: 90

See the configs/ directory for a complete, commented YAML per source kind.

Troubleshooting #

The XR session fails to create — the viewer launches the CloudXR runtime itself; check ~/.cloudxr/logs/cxr_server.*.log and runtime_stderr.log for the startup failure. If a runtime is already running from another app, pass --no-launch-cloudxr-runtime. Pass --mode window to render to a desktop window instead (no runtime involved).
No window appears over SSH — --mode window needs a local display; run on the machine you’re sitting at, or use a video-capable remote desktop.
“video source: no such file” — relative path: values resolve against the YAML’s directory (configs/), not the directory you launched from.
A source fails asking for CuPy / CUDA — check nvidia-smi works and setup completed; all sources allocate their frame buffers on the GPU.
Split mode renders nothing — check the sender is up (./camera_viz.sh service-status), $STREAMING_HOST was the workstation’s IP at deploy time, and UDP ports (default 5000+) aren’t firewalled.
Not sure which side is stuck? — set verbose: true at the top of the YAML for periodic per-source breadcrumbs on both ends.

How it works #

The sample is organized so that capture, transport, and visualization are cleanly separated, with Televiz as the compositor at the end of the chain:

camera_viz/
├── camera_viz.sh        — CLI: setup / loopback / run / deploy / service-*
├── camera_viz.py        — receiver / viewer (drives a Televiz VizSession)
├── camera_streamer.py   — robot-side RTP sender (per-camera supervisor)
├── pipeline/            — source ABC + threaded runner
├── placements/          — XR lock-mode strategies (world / head / lazy / gimbal)
├── sources/             — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264
├── transports/          — RTP sender + receiver (native + GStreamer)
├── codec/               — native NVENC / NVDEC pybind module
├── configs/             — one YAML per source kind
├── test_data/           — sample replay clip (Git LFS)
└── scripts/             — installer + systemd unit template

Sources (sources/) implement a common source ABC and hand frames to a threaded runner in pipeline/. Each source produces GPU frames where possible — e.g. the ZED source uses retrieve_image(MEM.GPU) so BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip.
The viewer (camera_viz.py) creates a VizSession and adds one QuadLayer per enabled camera, then submits each frame to its layer and calls render() once per frame. Stereo cameras submit both eyes.
Transport (transports/) carries the split mode: an RTP H.264 sender on the robot and a receiver on the workstation, with a native NVENC/NVDEC codec module (codec/) or a GStreamer fallback.
Placement (placements/) holds the XR lock-mode strategies. Placement is application policy — Televiz only renders a layer at whatever pose the app sets each frame.