Architecture

Isaac Capture provides shared capture infrastructure for teleoperation and egocentric data. It is a library used by applications and integration environments, including Isaac Lab and Sim, Isaac ROS, and other ROS stacks.

Isaac Capture components, shared infrastructure, and integration environments.

Capture components

Isaac Teleop provides robot-in-the-loop capture. Isaac Teleop introduces its workflows, including Teleop Session for connecting input devices, retargeting, and robot control. Remote Assist is also part of the teleoperation component; the Out-of-Band Teleop Control reference documents its control transport, rather than a complete operator workflow.

The Ego Pipeline covers human and egocentric data capture. Ego Capture and Ego Post-Processing are its two stages. They share the underlying device and data infrastructure with Isaac Teleop.

Shared infrastructure

Device plugins

The Spatial Trackers and Device Data interfaces provide typed access to device data. Plugins connect hardware such as gloves, body trackers, and cameras through a common transport. XR headsets provide poses and input through OpenXR. See Add a New Device for plugin development and Ecosystem for individual device capabilities.

Retargeters

The Retargeting Interface framework maps device inputs to robot embodiments through graph-based source nodes and retargeters. Hand retargeters and body-tracking inputs can be combined with application-specific mappings. The framework is shared; Isaac Teleop uses it to produce robot actions.

Data utilities

Shared FlatBuffers schemas define device data. MCAP Recording & Replay covers recording and playback; LeRobot and SO-101 covers LeRobot data collection workflows.

Visualization and streaming

Televiz (isaaccapture.viz) composites camera and sensor feeds and 3D content for XR headsets or desktop windows. It supports per-eye stereo views and GPU frame submission, and can share an OpenXR session with device tracking. See Camera Streaming for streaming camera data and CloudXR Service for the CloudXR service.

Integration and distribution

Applications use Isaac Capture in simulation and on real robots. Integration guides are organized under Isaac Lab & Sim and Isaac ROS; the ecosystem also includes research and custom ROS integrations. Integration names in the diagram do not imply that every workflow or device is supported.

The source is distributed through GitHub, and Python packages through PyPI. See Build from Source for a source build.