<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 Avatar SDK contributors. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

<a id="sharpa-avatarexo-glove"></a>

# Sharpa AvatarExo Glove

A Linux-only plugin for integrating Sharpa AvatarExo (AE) Gloves into the
Isaac Capture framework. The gloves stream IMU and encoder data wirelessly
through a USB dongle and the official Avatar SDK. The plugin maps that stream
into Isaac Capture’s usual surfaces: OpenXR hand tracking for the HUMAN skeleton,
`JointStateOutput` tensors for the RAW and ROBOT 22-DoF joint sets, and
inbound per-finger vibration on a haptic collection.

Any downstream consumer that already reads OpenXR hands (`HandsSource`) or
schema joint-state collections can use Sharpa AE Gloves through the same DeviceIO
interfaces. The plugin talks to the gloves directly, so do not run
`avatar-backend` or Avatar Desktop at the same time.

Plugin sources and the operator README live under
[src/plugins/sharpa_avatar](https://github.com/NVIDIA/IsaacCapture/tree/main/src/plugins/sharpa_avatar). A TeleopSession visualization example
is [examples/sharpa_avatar](https://github.com/NVIDIA/IsaacCapture/tree/main/examples/sharpa_avatar).

> ##### On this page
> 
> * [Data flow](#data-flow)
> * [Prerequisites](#prerequisites)
> * [Installation](#installation)
>   * [USB access (one-time, on the host)](#usb-access-one-time-on-the-host)
> * [Running the plugin](#running-the-plugin)
> * [Datasets](#datasets)
>   * [HUMAN (OpenXR hands)](#human-openxr-hands)
>   * [RAW and ROBOT (joint-state tensors)](#raw-and-robot-joint-state-tensors)
>   * [Haptic](#haptic)
>   * [Health reporting](#health-reporting)
> * [Wrist pose](#wrist-pose)
> * [Troubleshooting](#troubleshooting)
> * [License](#license)

<a id="data-flow"></a>

## Data flow

The glove SDK exposes three datasets plus haptics. HUMAN landmarks are placed
at a fused wrist pose and injected as OpenXR hands; RAW and ROBOT stay as
named joint-state collections:

```text
Sharpa AvatarExo Gloves (wireless via USB dongle)
        │
        ▼
avatar_hand_plugin  (Avatar SDK, production package)
        │
        ├── HUMAN  ──► OpenXR /hand/left, /hand/right   (26 joints)
        ├── RAW    ──► avatar_raw_left / avatar_raw_right
        ├── ROBOT  ──► avatar_robot_left / avatar_robot_right
        └── haptic ◄── avatar_glove_haptic  (five finger vibration motors)
```

#### SEE ALSO
[Retargeter: Manus to Sharpa](../../references/retargeting/sharpa.md) maps OpenXR hand tracking onto a
Sharpa robot hand. Avatar HUMAN output is the same OpenXR layer that
retargeter consumes.

<a id="prerequisites"></a>

## Prerequisites

- **Linux x86_64** (tested on Ubuntu 22.04).
- **Pinned production Avatar SDK** (`avatar-sdk` 1.7.3-17) from Sharpa’s
  signed APT repository. `install_avatar_sdk.sh` retrieves it; do not substitute
  `avatar-sdk-dev` or `avatar-sdk-beta`.
- **Sharpa AvatarExo Gloves** powered and paired with the USB wireless dongle.
- **CloudXR runtime** on the plugin host, unless a `TeleopSession` example
  launches it for you.
- A built Isaac Capture checkout. CMake 3.24 or newer is required.

The SDK stays external: headers, libraries, and runtime assets are not copied
into this repository or the plugin install prefix.

<a id="installation"></a>

## Installation

Install the pinned production SDK first if it is not already present:

```bash
./src/plugins/sharpa_avatar/install_avatar_sdk.sh
```

Then use the canonical top-level CMake workflow from the Isaac Capture root.
`-DAVATAR_SDK_ROOT` selects a non-default SDK tree; that tree must already
be complete:

```bash
cmake -B build \
   -DBUILD_PLUGIN_SHARPA_AVATAR=ON \
   -DAVATAR_SDK_ROOT=/opt/avatar-sdk
cmake --build build --parallel
cmake --install build
```

`-DBUILD_PLUGIN_SHARPA_AVATAR=ON` without a usable SDK under the selected
root skips this plugin and the rest of Isaac Capture still configures. See
[Build from Source](../../development/build_from_source/index.md).

<a id="usb-access-one-time-on-the-host"></a>

### USB access (one-time, on the host)

udev rules must run **on the host**, not inside a container:

```bash
./src/plugins/sharpa_avatar/install_udev_rules.sh
# then unplug and reconnect the glove
```

<a id="running-the-plugin"></a>

## Running the plugin

The plugin publishes through the CloudXR / OpenXR runtime. Start the runtime
and source its environment in the shell that launches the plugin:

```bash
python -m isaaccapture.cloudxr.service start --accept-eula   # runs in the background
source ~/.cloudxr/run/cloudxr.env
./install/plugins/sharpa_avatar/avatar_hand_plugin
```

See [CloudXR Service](../../references/cloudxr.md#dedicated-cloudxr-runtime) and
[Load CloudXR environment variables](../../references/cloudxr.md#load-cloudxr-environment-variables) for the full service setup.

`TeleopSession` can launch the same binary through `plugin.yaml`. The
visualization example does that for you (CloudXR and the plugin start unless
you opt out). Run it from the repository root, in a venv set up as in
[step 4 of Build from source](../../development/build_from_source/index.md#build-from-source-install-package):

```bash
uv pip install -e ./examples/sharpa_avatar --no-sources
python -m isaaccapture_examples.sharpa_avatar
```

Gloves reconnect from `update()` if they drop offline. Only one process
may hold the Avatar SDK at a time.

<a id="datasets"></a>

## Datasets

By default every path is on. Restrict them with `--datasets=`
(comma-separated: `human`, `raw`, `robot`, `haptic`):

```bash
./install/plugins/sharpa_avatar/avatar_hand_plugin --datasets=human,raw,robot,haptic
./install/plugins/sharpa_avatar/avatar_hand_plugin --datasets=human
```

| Device or collection                      | Contents                                          |
|-------------------------------------------|---------------------------------------------------|
| `/hand/left`, `/hand/right`               | OpenXR 26-joint poses from Avatar HUMAN landmarks |
| `avatar_raw_left`, `avatar_raw_right`     | RAW 22-DoF encoder joint state                    |
| `avatar_robot_left`, `avatar_robot_right` | ROBOT 22-DoF retargeted joint state               |
| `avatar_glove_haptic`                     | Inbound per-finger vibration commands             |

<a id="human-openxr-hands"></a>

### HUMAN (OpenXR hands)

HUMAN landmarks are mapped onto the 26 OpenXR hand slots (unsupported slots
stay invalid) and pushed with `HandInjector`. Hosts read them through
`HandsSource` like any other hand-tracking plugin.

<a id="raw-and-robot-joint-state-tensors"></a>

### RAW and ROBOT (joint-state tensors)

Each side pushes a `JointStateOutput` (tensor id `joint_state`) with the
22 named DoF from the pinned SDK. Consume them with `JointStateSource`
and the collection ids above.

<a id="haptic"></a>

### Haptic

The plugin reads `HapticCommand` on `avatar_glove_haptic` and drives
five finger vibration motors. The visualization example uses the same
per-finger pinch path as the generic glove haptic sample.

<a id="health-reporting"></a>

### Health reporting

The plugin reports the USB dongle, each physical glove, and each dataset as
separate devices. Physical-glove health combines SDK inventory, live ingress,
firmware diagnostics, and haptic errors. HUMAN, RAW, and ROBOT then report their
own freshness and fetch errors, so one healthy stream cannot hide another
stalled stream. Outputs disappear after ingress stops and are recreated when
the glove resumes; existing DeviceIO consumers bind to the new generation.

<a id="wrist-pose"></a>

## Wrist pose

Sharpa AE Glove data is wrist-relative, so the plugin anchors HUMAN landmarks
with a pose from `WristPoseSource` before publishing OpenXR hands. It prefers
optical hand tracking and falls back to the controller aim pose when needed.
If no wrist pose is available yet, the plugin still publishes a local skeleton
with VALID-only flags; TRACKED is set once the selected wrist source is actively
tracked.

The example places that local skeleton into a stable left/right display
frame. Pass `--world-frame` to leave the OpenXR poses unmodified.

<a id="troubleshooting"></a>

## Troubleshooting

| Symptom                          | Fix                                                                                                                                                               |
|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Avatar SDK installation fails    | Confirm access to Sharpa’s production APT endpoint and re-run<br/>`install_avatar_sdk.sh`.                                                                        |
| CMake skipped the plugin         | The SDK is missing under the selected root. Run<br/>`install_avatar_sdk.sh` and reconfigure with<br/>`-DBUILD_PLUGIN_SHARPA_AVATAR=ON`.                           |
| CMake rejected the Avatar SDK    | Install the pinned production package, or point `AVATAR_SDK_ROOT`<br/>at a complete SDK tree. A `-dev` / `-beta` package at<br/>`/opt/avatar-sdk` is not the pin. |
| USB dongle is not detected       | Run `install_udev_rules.sh` on the host, then unplug and reconnect<br/>the USB dongle.                                                                            |
| Plugin binary is not found       | Build and install through the top-level CMake workflow. The example looks under<br/>`install/plugins/sharpa_avatar/`.                                             |
| HUMAN / RAW / ROBOT stay offline | Power the gloves, and stop Avatar Desktop, `avatar-backend`, or<br/>any other process using the Avatar SDK.                                                       |
| Viser shows an empty grid        | HUMAN must be enabled (default). RAW-only is joint angles, not a<br/>3D skeleton. Confirm the example is not launched with<br/>`--datasets` that omit `human`.    |
| No vibration                     | Keep `haptic` in `--datasets` and close a fingertip toward the<br/>thumb (example pinch path).                                                                    |
| CloudXR runtime errors           | Check `python -m isaaccapture.cloudxr.service status`, and source<br/>`~/.cloudxr/run/cloudxr.env` in the same terminal as the plugin.                            |

<a id="license"></a>

## License

Plugin sources are Apache-2.0. The Avatar SDK is proprietary to Sharpa and
is subject to its own license; this project does not redistribute it.
