Camera Streaming#

camera_viz is the reference camera-streaming sample built on Televiz (isaacteleop.viz). It captures frames from one or more cameras and streams them to an XR headset — one plane per camera, aspect-fit — either directly or from a robot to a workstation over the network (split mode). For development and debugging it can also replay a video file in place of a camera, and render to a desktop window instead of the headset.

The sample lives at examples/camera_viz/. This page walks you from setup and a hardware-free first run to real cameras and the robot → workstation split mode. For the exact command surface and flags, see the README.

camera_viz in window mode with synthetic multi-camera feeds — `camera_viz` in XR mode — one plane per camera.#

Requirements #

A workstation meeting the system requirements (Ubuntu, NVIDIA GPU, CUDA driver) — every source hands frames to the renderer GPU-resident via CuPy.
For the default XR mode, a running CloudXR server with a connected headset — follow the quick start steps 3. Configure CloudXR (optional) and 5. Connect an XR headset. No headset handy? --mode window renders to a desktop window instead and only needs a local display.

Setup #

Clone the repository if you haven’t already (quick start step 1. Check out code base (for examples)), then run the sample’s one-time setup:

examples/camera_viz/camera_viz.sh setup
source examples/camera_viz/.venv/bin/activate

There is no need to install the isaacteleop pip package yourself — setup creates the sample’s own environment: it installs isaacteleop (which bundles Televiz) and every other Python dependency from PyPI into .venv/ via uv, builds the native NVENC/NVDEC codec, and probes system packages (GStreamer plugins, cairo / girepository headers, JetPack cuda-nvrtc + ld.so wiring). When something is missing it prints the exact apt-get line and prompts [y/N] — answering n or running non-interactively aborts.

By default setup provisions everything except ZED support; flags trim or extend that:

Flag	Effect
`--no-v4l2`	Skip USB / UVC webcam support (`opencv-python`).
`--no-oakd`	Skip OAK-D support (`depthai`).
`--no-rtp`	Skip split-mode dependencies: the GStreamer system packages and the native NVENC/NVDEC codec build. Direct mode still works.
`--with-zed`	Also build + install the ZED SDK’s Python API (`pyzed`). Requires the ZED SDK on the machine (default `/usr/local/zed`; override with `--zed-sdk PATH`).
`--sender-only`	Split mode only — robot-side install of just the sender’s dependencies.
`--jetson`	Split mode only — extra CUDA wiring JetPack images need on the robot.
`--venv PATH`	Install into an existing virtual environment instead of creating `.venv/`.
`--wheel PATH`	Install a locally built `isaacteleop` wheel instead of the PyPI release — for developing Isaac Teleop itself (see Build from Source).

First run — no camera required #

The video-replay source (type: video) plays a recording through exactly the same path a live camera uses, so it doubles as the quickest end-to-end check and as a stand-in feed while the real camera isn’t available. A test clip ships with the repo and configs/replay.yaml already points at it:

cd examples/camera_viz
./camera_viz.sh run configs/replay.yaml                 # XR headset (default)
./camera_viz.sh run configs/replay.yaml --mode window   # desktop window instead

You should see the terminal report the session and the source coming up:

camera_viz: source=local, mode=xr, xr=True, 1 layer(s)
[video] opening...
[video] connected
[video] streaming

and the clip looping on a plane in the headset — or in a desktop window (mode=window, xr=False) with the --mode window override.

To replay your own video, set a custom path: in configs/replay.yaml <examples/camera_viz/configs/replay.yaml> — relative paths resolve against the YAML’s directory. loop: false holds the last frame instead of rewinding; stereo: true replays a side-by-side recording (e.g. from a ZED) as stereo, splitting each frame into per-eye views (direct mode only).

Supported sources #

The source kind is selected by the type field of each entry in the YAML cameras list:

`type:`	Notes
`v4l2`	USB / UVC cameras — anything `v4l2-ctl --list-formats-ext` reports.
`oakd`	OAK-D mono RGB / LEFT / RIGHT (stereo not yet wired).
`zed`	ZED 2 / Mini / X One; mono or `stereo: true` (per-eye SDK retrieve, zero-copy on the GPU).
`video`	Video-file replay (anything OpenCV’s FFmpeg backend reads). Loops by default; `stereo: true` splits side-by-side recordings into eyes (viewer only).
`synthetic`	Debugging tool — GPU-generated test pattern, no hardware or file.

Running with a real camera #

Attach the camera to the machine that runs the viewer, keep source: local in the config, and run with the matching config:

./camera_viz.sh run configs/v4l2.yaml     # or oakd.yaml / zed.yaml

You should see the same startup lines as above with the camera’s tag ([v4l2], [oakd], [zed]) and the live feed. Multiple entries in the cameras list render as one plane each.

Display modes #

XR is the default: each camera renders as its own plane in the headset via the active OpenXR runtime, and stereo sources (stereo: true) render true side-by-side stereo. Pass --mode window (or set display.mode: window in the YAML) to render to a desktop window instead — no headset or runtime needed; stereo shows the left eye.

In XR, how a plane follows the operator’s head is the per-camera lock_mode under display.placements.<name>:

Mode	Behavior
`world`	Placed once in front of you and stays put.
`head`	Follows your head every frame.
`lazy`	World-locked, but re-snaps in front of you when you look away (default).

Lazy-mode knobs live under placements.<name>: look_away_angle_deg, reposition_distance, reposition_delay_s, transition_duration_s.

Split mode — robot → workstation over RTP #

Split mode runs the capture side on the robot (camera_streamer.py) and ships RTP H.264 to the viewer on the workstation (source: rtp).

Warning

Split mode is not recommended in most cases. It exists for one situation: the cameras are on the robot, but Isaac Teleop runs on a workstation, so the frames must be streamed to where Isaac Teleop is running. That costs a full extra encode/decode hop — NVENC on the robot, UDP, NVDEC on the workstation — so whenever a camera can attach directly to the machine running Isaac Teleop, run direct mode instead. Wired networks only: there is no retransmit or FEC — one lost packet corrupts one frame until the next IDR (default every 5 s).

In split mode every camera entry must pin width, height, and fps in the YAML — the receiver sizes its decoder from the config, not from the wire.

Set source: rtp in the config, export the robot/streaming credentials once per shell, then deploy the sender and run the viewer:

export REMOTE_HOST=10.0.0.5 REMOTE_USER=nvidia
export STREAMING_HOST=10.0.0.42                  # workstation IP

./camera_viz.sh deploy configs/v4l2.yaml         # full deploy + systemd unit on the robot
./camera_viz.sh run    configs/v4l2.yaml         # viewer on the workstation

deploy rsyncs the source to the robot, installs sender dependencies, renders a camera-streamer.service systemd user unit (injecting --host from $STREAMING_HOST without editing the YAML on disk), and enables it. Operate the running unit with ./camera_viz.sh service-{status,logs,restart}. The sender retries forever across unplug, SDK errors, and network blips.

Loopback #

Loopback is a testing / debugging aid, not a deployment mode: ./camera_viz.sh loopback configs/v4l2.yaml runs the sender and viewer together on 127.0.0.1 — the quickest way to smoke-test the RTP path on one machine. It also works camera-free with a mono type: video entry (set width / height / fps).

Configuration #

A single YAML drives both capture and visualization. Each entry in the cameras list becomes its own plane (and, in split mode, its own RTP port). Abbreviated:

source: local | rtp
streaming:
  host: 192.168.1.100         # workstation IP (overridden at deploy time)
encoder: auto | native | gstreamer

cameras:
  - name: cam
    enabled: true
    type: v4l2                # v4l2 | oakd | zed | video | synthetic
    width: 2560               # video: optional — defaults to the file's size
    height: 720               # (required when source: rtp)
    fps: 30
    stereo: false             # zed / video / synthetic — per-eye capture + SBS in XR
    path: clip.mp4            # video only — file to replay, relative to this YAML
    loop: true                # video only — rewind at end of file
    rtp:
      port: 5000              # left eye when stereo
      port_right: 5001        # required when stereo + source: rtp
      bitrate_mbps: 15

display:
  mode: xr | window           # default: xr
  window: { width, height }
  xr:     { near_z, far_z }
  clear_color: [r, g, b, a]
  placements:
    cam:
      lock_mode: lazy         # world | head | lazy
      distance: 1.5
      # size: [w_m, h_m]
      # stereo_baseline_mm: 0

See the configs/ directory for a complete, commented YAML per source kind.

Troubleshooting #

The XR session fails to create — the default mode needs the CloudXR server running and a headset connected (quick start steps 3. Configure CloudXR (optional) and 5. Connect an XR headset); pass --mode window to render to a desktop window instead.
No window appears over SSH — --mode window needs a local display; run on the machine you’re sitting at, or use a video-capable remote desktop.
“video source: no such file” — relative path: values resolve against the YAML’s directory (configs/), not the directory you launched from.
A source fails asking for CuPy / CUDA — check nvidia-smi works and setup completed; all sources allocate their frame buffers on the GPU.
Split mode renders nothing — check the sender is up (./camera_viz.sh service-status), $STREAMING_HOST was the workstation’s IP at deploy time, and UDP ports (default 5000+) aren’t firewalled.
Not sure which side is stuck? — set verbose: true at the top of the YAML for periodic per-source breadcrumbs on both ends.

How it works #

The sample is organized so that capture, transport, and visualization are cleanly separated, with Televiz as the compositor at the end of the chain:

camera_viz/
├── camera_viz.sh        — CLI: setup / loopback / run / deploy / service-*
├── camera_viz.py        — receiver / viewer (drives a Televiz VizSession)
├── camera_streamer.py   — robot-side RTP sender (per-camera supervisor)
├── pipeline/            — source ABC + threaded runner
├── placements/          — XR lock-mode strategies (world / head / lazy)
├── sources/             — V4L2 / OAK-D / ZED / video replay / synthetic / rtp_h264
├── transports/          — RTP sender + receiver (native + GStreamer)
├── codec/               — native NVENC / NVDEC pybind module
├── configs/             — one YAML per source kind
├── test_data/           — sample replay clip (Git LFS)
└── scripts/             — installer + systemd unit template

Sources (sources/) implement a common source ABC and hand frames to a threaded runner in pipeline/. Each source produces GPU frames where possible — e.g. the ZED source uses retrieve_image(MEM.GPU) so BGRA8 stays in VRAM and a CUDA kernel channel-swaps into contiguous RGBA with no host round-trip.
The viewer (camera_viz.py) creates a VizSession and adds one QuadLayer per enabled camera, then submits each frame to its layer and calls render() once per frame. Stereo cameras submit both eyes.
Transport (transports/) carries the split mode: an RTP H.264 sender on the robot and a receiver on the workstation, with a native NVENC/NVDEC codec module (codec/) or a GStreamer fallback.
Placement (placements/) holds the XR lock-mode strategies. Placement is application policy — Televiz only renders a layer at whatever pose the app sets each frame.