Retargeters: SO-101 (5-DOF arm)#

The SO-101 is a low-cost 5-DOF arm with a single-jaw gripper. The controller is meant to feel like holding the leader arm: the gripper pose follows the controller pose directly. A full SE3 pose is commanded and a single differential IK solves all 5 arm joints, tracking position exactly and orientation best-effort (a 5-DOF arm is over-determined by one DOF on a 6-DOF pose). These retargeters provide the pieces for comfortable XR controller teleoperation of the arm (used by the Isaac Lab Isaac-Stack-Cube-SO101-IK-Abs-v0 cube-stacking task):

SO101ClutchRetargeter – absolute EE pose (position + orientation) clutched as a full pose: both the home position and the home orientation are re-latched on every engage, so there is no teleport when the operator engages.
SO101GripperRetargeter – proportional (analog) jaw closedness from the controller trigger.

Together they flatten (via TensorReorderer) into an 8-D action [pos_x, pos_y, pos_z, quat_x, quat_y, quat_z, quat_w, gripper].

At a glance#

Retargeter	Output	What it does
`SO101ClutchRetargeter`	7-D `ee_pose` (xyz + xyzw quat)	Same output contract as `Se3AbsRetargeter`, but clutches the full pose: re-latches both the home position and the home orientation on every engage, composing the orientation delta on the left (base frame) with no fixed offset. Engages on `RUNNING` and `squeeze > squeeze_threshold`.
`SO101GripperRetargeter`	1 float `gripper_command` `c` in `[0, 1]`	Trigger -> jaw closedness (`0` = open, `1` = closed), with a released-end deadzone.

Why a clutch#

Se3AbsRetargeter maps the controller’s absolute pose straight to the EE target, so engaging teleop teleports the arm to wherever the controller happens to be. SO101ClutchRetargeter instead latches an engage origin and drives the EE from the delta relative to it, keeping position-control IK (use_relative_mode=False): it emits an absolute target, just rebased.

Each engaged frame emits:

pos = home_pos + position_scale * (grip_pos - origin_pos)
rot = (R_ctrl @ R_origin^-1) @ R_home

On the engage frame grip_pos == origin_pos and R_ctrl == R_origin, so the output is exactly the home pose whatever the scale – no teleport. Because the orientation delta is left-composed, a hand rotation about base Z maps to an EE rotation about base Z.

Frames#

The controller stream reaching SO101ClutchRetargeter must already be expressed in the robot base frame. In Isaac Lab the IsaacTeleopDevice rebases it upstream via its target_frame_prim_path (set to the robot base), composing base_T_world onto the XR anchor before the controllers are transformed; standalone graphs use ControllersSource.transformed. The clutch applies the delta directly, with no world->base rotation of its own.

Under an upstream rigid rebase p -> R @ p + t the rebase’s translation t cancels exactly in grip_pos - origin_pos. Its rotation R does not: it rotates the translation delta and conjugates the orientation delta, so a wrong rebase rotation mis-maps both channels – a hand push along +X can come out as EE motion along +Y. That rebase is static configuration and nothing in the retargeter can validate it.

Engaging#

Engagement is execution_state == RUNNING and squeeze > squeeze_threshold. The two conjuncts are different things and both are load-bearing: squeeze is operator intent, while execution_state is system readiness. An owning application that is still homing the arm should hold STOPPED so a squeeze cannot latch a home the arm has not reached yet – which also makes it safe to supply the real home late, via set_home_base_T_ee(), when the retargeting graph must be built before the arm is homed.

Latching uses a pending-latch sentinel: the latch is owed until a frame arrives that can be trusted, and it is re-armed on every disengaged, dropped, invalid or degenerate frame. Both halves matter. Re-arming on dropped frames in particular is what prevents a jump when the operator releases and re-squeezes across a tracking dropout – the squeeze is unobservable inside the gap, so without it the clutch would rebase against the previous engagement’s origin and command the whole accumulated hand motion at once.

Owning loops should read is_engaged rather than re-deriving engagement from the squeeze: its rising edge is the latch frame, including the cases where the latch was deferred by an unusable frame the loop cannot observe.

The home#

home_base_T_ee is a required base_T_ee 4x4 transform and both blocks are used: the translation seeds the home position, the rotation seeds the home orientation. Its rotation block must be a proper rotation (orthonormal, det == +1); a scaled or reflected block would still normalize to a unit quaternion and command a quietly wrong orientation, so it is rejected at construction.

Home latching on each engage is deliberately asymmetric: the home position comes from the arm’s measured pose when MEASURED_BASE_T_EE_INPUT is connected (so an arm that sagged while disengaged is not snapped back to a stale command), while the home orientation is always the last commanded rotation. The 5-DOF wrist tracks orientation only softly, so latching the measured orientation would inject that tracking offset into the command on every re-clutch.

MEASURED_BASE_T_EE_INPUT is declared OptionalType and a consumer may omit it entirely from connect() – optional inputs are exempt from the missing-input check. A consumer whose owning task slews the arm to a known configured home (Isaac Lab) legitimately leaves it unwired and rides the last-commanded fallback; that is a designed path, not degraded mode, so it is silent.

What OptionalType does not buy is the right to skip a key you did wire. When the input is connected to a ValueInput leaf, that leaf is an external graph input, and TeleopSession validates every external leaf name is present in external_inputs on every step, independently of OptionalType. Such a producer must send the key unconditionally, carrying an absent OptionalTensorGroup on frames with no pose.

Reset#

On execution_events.reset the held pose is re-seeded from the configured home – the transform most recently given to the constructor or to set_home_base_T_ee() – and the latch is re-armed.

The seed is a static configured transform, never live arm state, and that is the whole point. The reset pulse can reach the retargeter on either side of the owner’s actual teleport (Isaac Lab’s record_demos.py fires it from a headset control event before the teleport and from a success condition after it), so anything read from the arm at reset time is stale on one of those orderings.

The owning task is expected to slew the arm to that same configured pose on reset – Isaac Lab’s robot init_state does exactly that – which is what makes the re-seed jump-free. An owner whose reset does not move the arm should keep the configured home up to date with set_home_base_T_ee(), or simply not fire reset.

“Never live arm state” scopes to the re-seed, not the whole frame. A frame carrying reset can also latch the clutch, and the re-seed does not change how that latch behaves: the normal engage rule applies, so on such a frame the home position comes from measured_base_T_ee when it is connected, not from the re-seed. That does not escape the ordering problem above – the grip validity flag and the squeeze qualify the controller pose, and say nothing about whether the arm has been teleported yet, so on a reset-before-teleport ordering that measurement is the previous episode’s arm state. No consumer wires both today; one that does should either not fire reset on a frame it also engages, or leave the measured input unwired. The re-seed’s own job is unaffected: it defines what the retargeter holds, and re-latches from, when nothing engages.

Note

There is deliberately no orientation_offset. It is not merely unnecessary, it is wrong: appending a fixed offset gives (R_ctrl . R_origin^-1 . R_home) . R_off, and on the engage frame R_ctrl == R_origin, so the output is R_home . R_off != R_home – breaking the no-teleport invariant. Worse, because R_home re-latches from the last commanded rotation, the error compounds as R_off, R_off^2, R_off^3 over successive re-clutches.

Note

Orientation wind-up is a real property of this algebra and is preserved intentionally: the commanded rotation is unbounded in SO(3) across repeated re-clutches, and wrist_roll sits in the null space of the 5-DOF position objective.

Also note that position_scale applies to translation only – the orientation delta is always 1:1, since a scaled rotation is disorienting and the 5-DOF wrist tracks orientation softly anyway. A value < 1 shrinks robot travel relative to hand travel, which is what lets a comfortable operator arm sweep (~0.7 m) stay inside the SO-101’s ~0.35 m reach.

Gripper#

SO101GripperRetargeter maps the analog trigger to a jaw closedness c in [0, 1] (0 = open, 1 = closed) with a small released-end deadzone, so a half-pressed trigger leaves the jaw half-closed. Downstream, an order-locked JointPositionActionCfg applies the affine joint = offset + scale * c mapping c onto the open/close joint angles. This is deliberately independent of the shared GripperRetargeter’s binary +1 = open / -1 = closed sign.

Use it from Python#

import numpy as np

from isaacteleop.retargeters import (
    SO101ClutchRetargeter,
    SO101GripperRetargeter,
    TensorReorderer,
)
from isaacteleop.retargeting_engine.deviceio_source_nodes import ControllersSource
from isaacteleop.retargeting_engine.interface import OutputCombiner, ValueInput
from isaacteleop.retargeting_engine.tensor_types import TransformMatrix

def build_so101_stack_pipeline(base_T_gripper_home: np.ndarray):
    controllers = ControllersSource(name="controllers")
    world_T_anchor = ValueInput("world_T_anchor", TransformMatrix())
    # The clutch requires a base-frame controller stream; rebase it upstream.
    xformed = controllers.transformed(world_T_anchor.output(ValueInput.VALUE))

    # home_base_T_ee is REQUIRED and both blocks are used. Pass the arm's pose at the reset
    # pose, or a placeholder plus a set_home_base_T_ee() call before the first RUNNING frame.
    clutch = SO101ClutchRetargeter(
        name="ee_pose",
        home_base_T_ee=base_T_gripper_home,
        input_device=ControllersSource.RIGHT,
    )
    # MEASURED_BASE_T_EE_INPUT is optional and deliberately left unwired here: the owning task
    # slews the arm to base_T_gripper_home, so the last-commanded home fallback is correct.
    connected_clutch = clutch.connect({
        ControllersSource.RIGHT: xformed.output(ControllersSource.RIGHT),
    })

    gripper = SO101GripperRetargeter(name="gripper", input_device=ControllersSource.RIGHT)
    connected_gripper = gripper.connect(
        {ControllersSource.RIGHT: xformed.output(ControllersSource.RIGHT)}
    )

    # Keep all 7 pose names and pass the full pose (xyz + quat) plus gripper through.
    ee_elements = ["pos_x", "pos_y", "pos_z", "quat_x", "quat_y", "quat_z", "quat_w"]
    reorderer = TensorReorderer(
        input_config={
            "ee_pose": ee_elements,
            "gripper_command": ["gripper_value"],
        },
        output_order=ee_elements + ["gripper_value"],
        name="action_reorderer",
        input_types={"ee_pose": "array", "gripper_command": "scalar"},
    )
    connected = reorderer.connect({
        "ee_pose": connected_clutch.output("ee_pose"),
        "gripper_command": connected_gripper.output("gripper_command"),
    })
    return OutputCombiner({"action": connected.output("output")})

See Build a Retargeting Pipeline for the general pipeline-builder pattern and Retargeting Interface for the full retargeting interface.

Validate#

The retargeters ship with sim-free unit tests (trigger/clutch math plus per-frame compute behavior):

$ ctest --test-dir build -R retargeting_test_so101_retargeters --output-on-failure