Ecosystem#
Unified Stack for Sim & Real Teleoperation#
A single framework that works seamlessly across simulated and real-world robots, ensuring streamlined device workflow and consistent data schemas.
Stack |
Description |
Status |
|---|---|---|
ROS2 |
Widely adopted middleware for robot software integration and communication |
Supported |
Isaac Sim |
Simulation platform to develop, test, and train AI-powered robots |
Supported (v6.0) |
Isaac Lab |
Unified framework for robot learning designed to help train robot policies |
Supported (v3.0) |
Isaac ROS |
NVIDIA CUDA-accelerated toolkit for ROS2 |
Planned |
Isaac Arena |
Isaac Lab extension for large-scale evaluation and resource orchestration |
Planned (v0.2) |
Supported Input Devices#
Isaac Teleop provides a standardized interface for teleoperation devices, removing the need for custom device integrations and ongoing maintenance. It supports multiple XR headsets and tracking peripherals. Each device provides different input modes, which determine which retargeters and control schemes are available. Easily extend support for additional devices through a plugin system; see Isaac Teleop device plugin (C++ level) for details.
Device |
Input Modes |
Client / Connection |
Notes |
|---|---|---|---|
Hand tracking (26 joints), spatial controllers |
Isaac XR Teleop Sample Client (visionOS app) |
Build from source; see Connect Apple Vision Pro |
|
Motion controllers (triggers, thumbsticks, squeeze), hand tracking |
Isaac Teleop Web Client (browser) |
||
Motion controllers, hand tracking |
Isaac Teleop Web Client (browser) |
Requires Pico OS 15.4.4U or newer |
|
Full body tracking |
Isaac Teleop Web Client (browser) |
Requires Pico OS 15.4.4U or newer
Requires Pico Browser 4.0.40 or newer (Enterprise enabled)
|
In addition to the fully integrated XR headsets, Isaac Teleop also supports standalone input devices. Those devices are typically directly connected to the workstation where the Isaac Teleop session is running via USB or Bluetooth. See Isaac Teleop device plugin (C++ level) for more details.
Device |
Input Modes |
Client / Connection |
|---|---|---|
High-fidelity finger tracking (Manus SDK) |
Manus Gloves Plugin (CLI tool) |
|
3-axis foot pedal |
Generic 3-axis Pedal Plugin (CLI tool) |
|
Offline data recording |
OAK-D Camera Plugin (CLI tool) |
Planned Input Device Support#
The following input devices and device categories are planned for support in the future:
Device |
Input Modes |
Client / Connection |
Status |
|---|---|---|---|
JoyLo |
Master Manipulators |
CLI tool with USB connection |
Planning, see #272 |
Gello |
Master Manipulators |
CLI tool with USB connection |
Planning, see #273 |
Master Manipulators |
CLI tool with USB connection |
Planning, see #274 |
|
SO-101 |
Master Manipulators |
CLI tool with USB connection |
Planning, see #275 |
HID input |
CLI tool with USB connection |
Planning, see #276 |
|
Gamepad |
HID input |
CLI tool with USB/Bluetooth connection |
Planning, see #277 |
Keyboard |
HID input |
CLI tool with USB/Bluetooth connection |
Planning, see #278 |
Targeted Robotics Embodiments#
Retarget the standardized device outputs to different embodiments.
Reference retargeter implementations, including popular embodiments such as Unitree G1.
Retargeter tuning UI to facilitate live retargeter tuning.
Device Acquisition#
For inquiries about acquiring supported or planned devices, please contact the manufacturers directly. Each device name in the tables above links to the corresponding manufacturer page.