Quick Start#

This guide walks through running a teleoperation session with an XR headset using CloudXR. By the end you will have the retargeting pipeline processing live hand/controller data and printing gripper commands to the terminal.

1. Check out code base (Optional)#

Clone the repository and enter the project directory:

git clone https://github.com/NVIDIA/IsaacTeleop.git
cd IsaacTeleop

As a quick start guide, we don’t need to build the code base from source. However, we still need to clone the repository for a couple quick samples to run.

2. Install the `isaacteleop` pip package #

In a new terminal, install the package from PyPI (or from a local wheel if you built from source):

# From PyPI
pip install 'isaacteleop[cloudxr,retargeters]~=1.0.0' --extra-index-url https://pypi.nvidia.com

Instead of installing the package from PyPI, you can build from source and install the local wheel. See Build from Source for more details.

3. Run CloudXR Server #

Start the CloudXR runtime. The first run downloads the CloudXR Web Client SDK and asks you to review and accept the EULA:

python -m isaacteleop.cloudxr

To bypass the interactive EULA prompt (e.g. for CI or headless runs), pass the flag:

python -m isaacteleop.cloudxr --accept-eula

You should see output similar to:

Important

Keep this terminal open — CloudXR must stay running for the duration of the session. Open a new terminal for the remaining steps.

Also take note of the source /home/dev/.cloudxr/run/cloudxr.env path it mentioned in the output. You will need to source it in step 6. Load CloudXR environment variables.

CloudXR Configurations (Optional)#

You can also inspect the CloudXR environment variables by running:

cat ~/.cloudxr/run/cloudxr.env

You should see:

export NV_DEVICE_PROFILE=auto-webrtc
...

By default, the CloudXR runtime is configured to use the auto-webrtc device profile for Pico & Quest headsets. For Apple Vision Pro, the runtime is configured to use the auto-native device profile.

To do that, you can override CloudXR configurations by creating an env file and pass it to the CloudXR runtime via the --cloudxr-env-config argument.

echo 'NV_DEVICE_PROFILE=auto-native' > custom.env
python -m isaacteleop.cloudxr --cloudxr-env-config=./custom.env

Again, you can also inspect the CloudXR environment variables by looking at the ~/.cloudxr/run/cloudxr.env file:

Custom CloudXR configurations

Custom CloudXR configurations#
Variable	Default	Description	Possible Values
NV_DEVICE_PROFILE	`auto-webrtc`	Custom device profile to use	`auto-webrtc` `auto-native` `Quest3` `AppleVisionPro`
NV_CXR_ENABLE_PUSH_DEVICES	`true`	Enable or disable push device overseer for hand tracking	`true` `false`
NV_CXR_FILE_LOGGING	`true`	Enable or disable file-based logging, when disabled logs are printed to the console	`true` `false`

export NV_DEVICE_PROFILE=auto-native
...

4. Whitelist ports for Firewall #

CloudXR requires certain network ports to be open. Depending on your firewall configuration, you might need to whitelist them manually.

Please see the CloudXR network setup for more details for other network configurations (such as running the CloudXR runtime and wss proxy in containerized environment; or using Vision Pro client).

5. Connect an XR headset #

6. Load CloudXR environment variables #

Open a new terminal and source the CloudXR environment variables posted from the CloudXR runtime in 3. Run CloudXR Server:

Source the setup script so that the OpenXR runtime points to CloudXR:

source ~/.cloudxr/run/cloudxr.env

Important

Make sure to run the rest of the commands in the same terminal. Or if have to open a new terminal, source the CloudXR environment variables again.

7. Run a teleop example #

Run the simplified gripper retargeting example. This demonstrates the full pipeline: reading XR controller input via CloudXR, retargeting it through the GripperRetargeter, and printing the resulting gripper command values:

python examples/teleop/python/gripper_retargeting_example_simple.py

Once running, squeeze the controller triggers on your XR headset to control the gripper. You should see periodic status output:

============================================================
Gripper Retargeting - Squeeze triggers to control grippers
============================================================

[  0.5s] Right: 0.00
[  1.0s] Right: 0.73
[  1.5s] Right: 1.00
...

The example runs for 20 seconds and then exits. To try other examples, see examples/teleop/python/ — for instance:

se3_retargeting_example.py — maps hand or controller poses to end-effector poses (absolute or relative)
dex_bimanual_example.py — bimanual dexterous hand retargeting
gripper_retargeting_example.py — full gripper example with more configuration options

Next steps #

Teleoperation in Isaac Lab

Follow instructions in Teleoperation and Imitation Learning with Isaac Lab Mimic to know more about how to collect demonstrations with Isaac Lab and how to augment them with Isaac Lab Mimic and train imitation learning policies.

If you are new to Isaac Lab, follow instructions in Isaac Lab Quick Start to get started.

Teleoperation with Isaac ROS

Check out the examples/teleop_ros2/ directory for an example on how to make a ROS 2 message publisher using Isaac Teleop.

We are also working on a Unitree G1-based end-to-end teleoperation, data collection, and imitation learning solution for ROS2 in an upcoming Isaac ROS release. Stay tuned!

ROS is a trademark of Open Robotics.

More Information #

Teleop Session — learn how TeleopSession works and how to build custom retargeting pipelines
Build from Source — build the C++ core, Python bindings, and plugins from source