<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

<a id="tactile-devices"></a>

# Tactile devices

Tactile devices capture contact measurements. For actuator output, see
[Haptic Feedback](../../references/haptic_feedback.md).

The common tactile interface separates each frame’s force measurements from the
device’s static layout. A *taxel* is one tactile sensing element: its force has
three components, and its layout describes where it is attached to a skeleton.
The current geometry mapping is linear blend skinning (LBS) over hand joints.

> ##### On this page
> 
> * [Data contract](#data-contract)
> * [Coordinate conventions](#coordinate-conventions)
>   * [Hand joints and the skeleton root](#hand-joints-and-the-skeleton-root)
>   * [Taxel frames](#taxel-frames)
>   * [Left and right hands](#left-and-right-hands)
> * [Mapping taxels onto the moving hand](#mapping-taxels-onto-the-moving-hand)
> * [Publishing and consuming](#publishing-and-consuming)
> * [Vendor integrations](#vendor-integrations)

<a id="data-contract"></a>

## Data contract

The contract is defined by
[src/core/schema/fbs/tactile_array.fbs](https://github.com/NVIDIA/IsaacCapture/blob/main/src/core/schema/fbs/tactile_array.fbs) and
[src/core/schema/fbs/tactile_metadata.fbs](https://github.com/NVIDIA/IsaacCapture/blob/main/src/core/schema/fbs/tactile_metadata.fbs):

| Type               | Meaning                                                                                                                                                                                                                    |
|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `TactileDataArray` | One force vector per taxel, in newtons, ordered exactly as the metadata.<br/>The Python view is an `(N, 3)` float32 array.                                                                                                 |
| `TactileMetadata`  | The static `geometry_mapping` and `segment_source` for that stream.<br/>Currently these select `LBS` and `HandSkeletonRef`.                                                                                                |
| `HandSkeletonRef`  | An ordered list of unique `HandJoint` identifiers from<br/>[src/core/schema/fbs/hand.fbs](https://github.com/NVIDIA/IsaacCapture/blob/main/src/core/schema/fbs/hand.fbs). `NUM_JOINTS` is a count,<br/>not a usable joint. |
| `LBS`              | `J` segment bind poses, `N` taxel bind poses, and four segment indices<br/>and weights per taxel. An index addresses the ordered segment list,<br/>rather than a `HandJoint` ordinal.                                      |

Python pose arrays have shape `(count, 7)`: position `(x, y, z)` in meters,
then a unit quaternion `(x, y, z, w)`. Identity orientation is `(0, 0, 0, 1)`.
All frames are right-handed. A pose maps coordinates from its local frame into
its reference frame: `p_reference = R(q) * p_local + position`.

<a id="coordinate-conventions"></a>

## Coordinate conventions

<a id="hand-joints-and-the-skeleton-root"></a>

### Hand joints and the skeleton root

`HandSkeletonRef` uses `HandJoint.WRIST` as the skeleton root. Its origin
is the wrist pivot. The joint names and local orientations follow
[OpenXR hand-joint conventions](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#convention-of-hand-joints):

| Root axis   | Anatomical direction                                                                                                      |
|-------------|---------------------------------------------------------------------------------------------------------------------------|
| `+Y`        | Out through the back of the hand at the wrist.                                                                            |
| `+Z`        | Along the line through the wrist and middle-finger metacarpal, pointing<br/>toward the forearm, away from the fingertips. |
| `+X`        | `+Y cross +Z`, completing the right-handed basis.                                                                         |

Finger-joint local `+Z` follows its bone away from the fingertip; local `+Y`
points toward the nail/back side in the fully open hand. The palm uses the
middle metacarpal for its longitudinal axis. These local frames move with the
joints when the hand bends; their axes are not fixed world directions.

`segment_bind_poses[j]` maps joint-local coordinates into the wrist frame in
the calibrated bind configuration. All segments share that root; the poses are
not relative to each joint’s parent. If WRIST appears in the segment list, its
bind pose is identity. `taxel_bind_poses` use the same root and bind
configuration. The bind configuration must match the device’s calibration;
the schema does not prescribe joint angles or a particular hand size.

**HandPose is not automatically wrist-relative.** The live `HandTracker`
copies OpenXR joint poses in the session base space, currently `STAGE`.
To obtain current joint poses in the tactile skeleton’s wrist frame, use the
wrist and joint poses from the same valid hand sample:

```text
T_root_joint(t) = inverse(T_base_wrist(t)) * T_base_joint(t)
```

Here `T_A_B` maps a point from frame B to frame A. This conversion uses both
the wrist translation and rotation. Subtracting the wrist position alone is
insufficient. A downstream consumer must perform this conversion when using
root-relative joint poses; `TactileSource` does not rebase a hand stream.

<a id="taxel-frames"></a>

### Taxel frames

A taxel pose’s origin is its sensing location. Its orientation defines:

| Taxel axis   | Direction                                                              |
|--------------|------------------------------------------------------------------------|
| `+Z = N`     | Unit normal pointing out of the sensing surface, away from the device. |
| `+X = T`     | A calibrated unit tangent perpendicular to `N`.                        |
| `+Y`         | `N cross T`. Thus `+X cross +Y = +Z`.                                  |

The producer chooses and calibrates `T` for each taxel, then stores the full
orientation in `taxel_bind_poses`. A normal alone cannot determine the tangent
direction. In particular, taxel `+Z` is a surface normal, whereas hand-joint
`+Z` follows a bone; the two local frames generally differ.

Forces `(fx, fy, fz)` are components in these taxel-local axes, rather than in
the wrist or world frame. Positive `fz` points outward; this does not imply
that a vendor’s positive compression reading can be copied directly. The schema
does not identify which body the reported force acts on. A device integration
must document that choice and calibrate its units, axes, and signs accordingly.
Raw pressure or ADC counts require conversion to force before publication.

<a id="left-and-right-hands"></a>

### Left and right hands

Both hands use the same joint identifiers, right-handed algebra, units,
quaternion ordering, and taxel normal/tangent rules. Hand side does not reverse
the cross product or automatically negate a force component. Anatomical geometry
and the calibrated poses can differ between hands. For example, with an open
hand, wrist `+X` points toward the thumb side on the left hand and the little
finger side on the right hand.

`HandSkeletonRef` carries neither a side nor the identity of a hand tracker.
The application must associate each tactile stream with the correct hand stream
and calibration, using configuration and distinct collection IDs such as
`tactile/left` and `tactile/right`. The runtime does not infer side from
these names. Mirroring positions alone is insufficient: a reflection cannot be
encoded as a rotation quaternion, so each hand needs a proper right-handed
orientation for every joint and taxel.

<a id="mapping-taxels-onto-the-moving-hand"></a>

## Mapping taxels onto the moving hand

For a taxel’s homogeneous bind position `p_bind` in the skeleton root, LBS
computes its current root-relative position as:

```text
p_root(t) = sum_k w[k] * C[j[k]](t) * inverse(B[j[k]]) * p_bind
```

`B[j]` is `segment_bind_poses[j]` and `C[j](t)` is the current pose of
`joint_ids[j]` in the current wrist frame. Each taxel has four slots; weights
must be finite, nonnegative, and sum to one (validation tolerance `1e-5`).
Every slot’s index must be valid, including slots with zero weight. A taxel
rigidly attached to one segment can use weights `(1, 0, 0, 0)` and repeat that
segment’s index in all four slots.

To express the resulting position in the hand tracker’s base space:

```text
p_base(t) = T_base_wrist(t) * p_root(t)
```

LBS specifies blended positions; it does not define a unique rigid taxel frame
after blending multiple segment rotations. A consumer that needs animated
normals or world-space forces must choose an orientation/normal mapping rule.
For a single rigid attachment, the taxel transform is unambiguous:
`C[j](t) * inverse(B[j]) * T_root_taxel_bind`. Transform force directions using
rotation only, without adding the pose translation.

<a id="publishing-and-consuming"></a>

## Publishing and consuming

A C++ plugin publishes serialized samples with `SchemaPusher`; a Python
producer can use `TensorPushTracker`. The paired streams are:

```text
<collection_id>           tensor "tactile"  -> TactileDataArray
<collection_id>/metadata  tensor "metadata" -> TactileMetadata
```

Publish metadata on startup and periodically so consumers that start later can
discover it. Publish the same encoded metadata throughout the tracker session.
`TactileTracker` validates and adopts the first metadata, records it once,
and exposes it through `get_metadata()`. It withholds force data until metadata
is confirmed. Different or invalid metadata causes an error; after a producer
restart, its metadata must be confirmed again. See [Trackers with metadata](../../references/generated_trackers.md#tracker-metadata).

The consumer’s `max_flatbuffer_size` must be at least the producer’s declared
sample size, including FlatBuffer overhead. The default is
`TactileTracker.DEFAULT_MAX_FLATBUFFER_SIZE`; pass a larger
`max_flatbuffer_size` to `TactileTracker` when the producer uses larger force
samples. Metadata has its own
`TRACKER_METADATA_MAX_SIZE` limit, including FlatBuffer overhead.
The data array length must match the metadata’s taxel count.

Applications can read `TactileTracker` directly for live or replay data.
Retargeting graphs can instead use `TactileSource(name, collection_id,
num_segments, num_taxels)` from
`isaaccapture.retargeting_engine.deviceio_source_nodes`. It emits
`taxel_forces` and `tactile_layout` arrays; constructor counts fix their
shapes. It adapts the frame and static layout to graph tensors; the application
supplies hand motion and evaluates the geometry mapping separately.

The runnable feeder, direct tracker consumer, and MCAP replay commands are in
[examples/tactile/README.md](https://github.com/NVIDIA/IsaacCapture/blob/main/examples/tactile/README.md). The feeder uses synthetic geometry to
exercise transport; it is not an anatomical calibration model. Live operation
requires an OpenXR runtime and an available HMD system, including when using a
headless session; replay does not require the runtime.

<a id="vendor-integrations"></a>

## Vendor integrations

The guides below describe individual devices. The existing OGLO integration
uses its own `OgloGloveSample` schema with raw measurements; it does not
automatically publish the common force-and-layout contract described above.

* [OGLO Tactile Glove Plugin](oglo.md)
