Device Trackers#

Trackers (defined in src/core/deviceio_trackers) are the consumer-side API for reading device data from an active DeviceIOSession. Each tracker manages one logical device, queries the OpenXR runtime every frame, and exposes the latest sample through typed get_*() accessors.

There are two categories of trackers:

OpenXR-direct trackers – read pose and input data through standard OpenXR APIs (xrLocateSpace, xrSyncActions, etc.):

SchemaTracker-based trackers – create new device type by defining a FlatBuffer schema and reading it from OpenXR tensor collections via the SchemaTracker utility.

All trackers follow the same lifecycle:

  1. Construct the tracker.

  2. Pass it (along with any other trackers) to DeviceIOSession::run().

  3. Call session.update() each frame.

  4. Read data with the tracker’s get_*() method.

Note

The DeviceIOSession is considered a low-level API. In practice, it is recommended to use the Teleop Session to manage a teleop session with multiple device trackers and retargeters to work together.

Data Schema Convention#

Every tracker’s data is defined by a FlatBuffers schema under src/core/schema/fbs. Each schema follows a three-tier convention:

// 1. Inner data table -- the actual payload
table Xxx {
    field_a: SomeType (id: 0);
    field_b: AnotherType (id: 1);
}

// 2. Tracked wrapper -- used by the in-memory tracker API.
//    data is null when the tracked entity is inactive.
table XxxTracked {
    data: Xxx (id: 0);
}

// 3. Record wrapper -- used as the MCAP recording root type.
//    Adds a DeviceDataTimestamp alongside the payload.
table XxxRecord {
    data: Xxx (id: 0);
    timestamp: DeviceDataTimestamp (id: 1);
}

root_type XxxRecord;

  • Inner data table (e.g. HeadPose, HandPose, ControllerSnapshot) – contains the device-specific fields. All fields are present when the parent wrapper’s data pointer is non-null.

  • Tracked wrapper (e.g. HeadPoseTracked) – wraps the inner data in an optional data field. The in-memory get_*() accessors return a reference to this wrapper. When data is nullptr (C++) or None (Python), the device is inactive or no sample has arrived yet.

  • Record wrapper (e.g. HeadPoseRecord) – wraps the inner data plus a DeviceDataTimestamp. This is the root_type written to MCAP channels by the recorder via serialize_all().

Shared Types#

DeviceDataTimestamp (src/core/schema/fbs/timestamp.fbs)

All timestamp fields are int64 nanoseconds.

Field

Description

available_time_local_common_clock

System monotonic time when the sample became available to the recording system. Useful for measuring pipeline latency.

sample_time_local_common_clock

System monotonic time when the sample was captured. Enables cross-device synchronization (values from different devices share the same clock domain).

sample_time_raw_device_clock

Timestamp from the device’s own clock. Values from different devices are not directly comparable.

Pose (src/core/schema/fbs/pose.fbs)

struct Point      { x: float; y: float; z: float; }
struct Quaternion  { x: float; y: float; z: float; w: float; }
struct Pose {
  position: Point;       // meters
  orientation: Quaternion;
}

Tracker Reference#

HeadTracker#

Tracks the HMD head pose via the OpenXR view space.

HandTracker#

Tracks articulated hand joints (26 joints per hand, following the OpenXR XrHandJointEXT ordering) using the XR_EXT_hand_tracking extension.

ControllerTracker#

Tracks both left and right controllers – grip and aim poses, plus button and axis inputs. Uses standard OpenXR action bindings.

FullBodyTracker#

Tracks 24 body joints through a vendor-selected backend. The tracker itself is a vendor-agnostic marker and carries no vendor or live/replay state: a live session picks the backend via VendorConfig (see Vendor Selection), and replay reads the recorded full_body channel regardless of which vendor produced it. When no vendor is selected, the default vendor body.pico-xr reads the PICO XR_BD_body_tracking extension directly.

Note

FullBodyTrackerPico remains available as a deprecated alias for FullBodyTracker so existing scripts run unchanged.

FrameMetadataTrackerOak#

Multi-channel tracker for per-frame metadata from OAK camera streams. Uses the SchemaTracker utility internally.

Generic3AxisPedalTracker#

Reads foot pedal axis values pushed by a device plugin through OpenXR tensor collections. Uses the SchemaTracker utility internally.

Note

The Python method is named get_pedal_data() (instead of the C++ get_data()).

Vendor Selection#

Some trackers are vendor-agnostic markers: the tracker declares what device data it represents, while a live session chooses which backend (“vendor”) produces that data. This mirrors how live-vs-replay is chosen at the session level – the same tracker instance works across vendors and across live and replay. FullBodyTracker is currently the only vendored tracker; its default vendor body.pico-xr reads the PICO XR_BD_body_tracking extension.

Select a vendor by passing a VendorConfig to both DeviceIOSession.get_required_extensions() and DeviceIOSession.run(). A VendorConfig maps tracker instances to a TrackerVendor(id, params), where id selects the backend from the live factory’s vendor registry and params carries free-form string key/value options for it. Trackers left out of the config use their default vendor. Vendor selections on non-vendored trackers, and unknown vendor ids, are rejected at session construction.

import isaacteleop.deviceio as deviceio

body = deviceio.FullBodyTracker()

# Select the backend for the vendored tracker (default shown explicitly).
vendor_config = deviceio.VendorConfig([
    (body, deviceio.TrackerVendor("body.pico-xr")),
])

required_extensions = deviceio.DeviceIOSession.get_required_extensions(
    [body], vendor_config
)
with deviceio.DeviceIOSession.run(
    [body], handles, None, vendor_config
) as session:
    ...

Replay is always vendor-neutral: the replay full-body impl reads the recorded full_body channel regardless of which live vendor produced it, so VendorConfig applies to live sessions only. The vendor registry is open for additional pre-built plugin vendors without changing the tracker marker.

When driving devices through the higher-level teleop session manager, vendor selection is carried on the DeviceIO source itself via its vendor argument (e.g. FullBodySource(name="full_body", vendor=deviceio.TrackerVendor("body.pico-xr"))), so it travels with the pipeline into both extension discovery and session construction; see Teleop Session.

Usage Examples#

For end-to-end usage patterns combining trackers with a DeviceIOSession, see:

For higher-level usage with the teleop session manager and retargeting, see: