Ecosystem#

Unified Stack for Sim & Real Teleoperation#

A single framework that works seamlessly across simulated and real-world robots, ensuring streamlined device workflow and consistent data schemas.

Robotics Stacks#

Stack

Description

Status

ROS2

Widely adopted middleware for robot software integration and communication

Supported

Isaac Sim

Simulation platform to develop, test, and train AI-powered robots

Supported (v6.0)

Isaac Lab

Unified framework for robot learning designed to help train robot policies

Supported (v3.0)

Isaac ROS

NVIDIA CUDA-accelerated toolkit for ROS2

Planned

Isaac Arena

Isaac Lab extension for large-scale evaluation and resource orchestration

Planned (v0.2)

Supported Input Devices#

Isaac Teleop provides a standardized interface for teleoperation devices, removing the need for custom device integrations and ongoing maintenance. It supports multiple XR headsets and tracking peripherals. Each device provides different input modes, which determine which retargeters and control schemes are available. Easily extend support for additional devices through a plugin system; see Isaac Teleop device plugin (C++ level) for details.

XR Headsets and Tracking Peripherals#

Device

Input Modes

Client / Connection

Notes

Apple Vision Pro

Hand tracking (26 joints), spatial controllers

Isaac XR Teleop Sample Client (visionOS app)

Build from source; see Connect Apple Vision Pro

Meta Quest 2/3/3S

Motion controllers (triggers, thumbsticks, squeeze), hand tracking

Isaac Teleop Web Client (browser)

See Connect Quest and Pico

Pico 4 Ultra

Motion controllers, hand tracking

Isaac Teleop Web Client (browser)

Requires Pico OS 15.4.4U or newer

Pico Motion Tracker

Full body tracking

Isaac Teleop Web Client (browser)
Requires Pico OS 15.4.4U or newer
Requires Pico Browser 4.0.40 or newer (Enterprise enabled)

In addition to the fully integrated XR headsets, Isaac Teleop also supports standalone input devices. Those devices are typically directly connected to the workstation where the Isaac Teleop session is running via USB or Bluetooth. See Isaac Teleop device plugin (C++ level) for more details.

Standalone Input Devices#

Device

Input Modes

Client / Connection

Manus Gloves

High-fidelity finger tracking (Manus SDK)

Manus Gloves Plugin (CLI tool)

Logitech Rudder Pedals

3-axis foot pedal

Generic 3-axis Pedal Plugin (CLI tool)

OAK-D Camera

Offline data recording

OAK-D Camera Plugin (CLI tool)

Planned Input Device Support#

The following input devices and device categories are planned for support in the future:

Planned Input Devices#

Device

Input Modes

Client / Connection

Status

JoyLo

Master Manipulators

CLI tool with USB connection

Planning, see #272

Gello

Master Manipulators

CLI tool with USB connection

Planning, see #273

Haply

Master Manipulators

CLI tool with USB connection

Planning, see #274

SO-101

Master Manipulators

CLI tool with USB connection

Planning, see #275

3D Space Mouse

HID input

CLI tool with USB connection

Planning, see #276

Gamepad

HID input

CLI tool with USB/Bluetooth connection

Planning, see #277

Keyboard

HID input

CLI tool with USB/Bluetooth connection

Planning, see #278

Targeted Robotics Embodiments#