LeRobot and SO-101#
With Isaac Teleop, we got an end-to-end data collection and training pipeline for SO-101 in both sim and real teleoperation.
The SO-101 is a low-cost, open-source robot arm that has become a popular platform in the LeRobot community. Isaac Teleop lets you drive an SO-101 from more than one teleoperation device to collect demonstrations — in simulation with Isaac Lab and on real hardware with SO-101 support in LeRobot — then train a GR00T N1.7 manipulation policy on the result and close the loop from sim to real.
The same SO-101 task runs in Isaac Lab and on the real arm, each recording a LeRobot dataset. The two panels below stack on narrow screens and sit side by side on wider ones, and can be replaced independently:
End-to-end workflow#
The same SO-101 embodiment runs in simulation and on real hardware, so a single workflow carries you from teleoperation to a deployed policy:
Teleoperate and collect. Drive the SO-101 with the XR controller or the SO-101 Leader and record demonstrations — in real and in simulation. Both produce datasets in the LeRobot format.
Data Collection with LeRobot.#
Train. Fine-tune a GR00T N1.7 policy on the collected dataset.
The trained GR00T N1.7 policy running autonomously on the SO-101.#
Deploy. Take the policy from sim to real — see the Sim-to-Real SO-101 learning path — addressing the sim-to-real gap with domain randomization, sim/real co-training, and actuator-gap compensation.
In this section#
Supported Teleop devices — the supported teleop devices: the XR controller and the SO-101 Leader.
Data Collection in Real — record demonstrations on a physical SO-101.
Data Collection in Sim — record demonstrations in Isaac Lab.
Model Training with GR00T — fine-tune a GR00T N1.7 policy on the collected data.
New to XR teleoperation? Start with the Isaac Teleop Quick Start to set up CloudXR and connect a headset.
Pending Tasks#
The guides below are still being written:
See also: the Sim-to-Real SO-101 learning path.