Device Trackers#

Trackers (defined in src/core/deviceio_trackers) are the consumer-side API for reading device data from an active DeviceIOSession. Each tracker manages one logical device, queries the OpenXR runtime every frame, and exposes the latest sample through typed get_*() accessors.

There are two categories of trackers:

OpenXR-direct trackers – read pose and input data through standard OpenXR APIs (xrLocateSpace, xrSyncActions, etc.):

SchemaTracker-based trackers – create new device type by defining a FlatBuffer schema and reading it from OpenXR tensor collections via the SchemaTracker utility.

All trackers follow the same lifecycle:

  1. Construct the tracker.

  2. Pass it (along with any other trackers) to DeviceIOSession::run().

  3. Call session.update() each frame.

  4. Read data with the tracker’s get_*() method.

Note

The DeviceIOSession is considered a low-level API. In practice, it is recommended to use the Teleop Session to manage a teleop session with multiple device trackers and retargeters to work together.

Data Schema Convention#

Every tracker’s data is defined by a FlatBuffers schema under src/core/schema/fbs. Each schema follows a three-tier convention:

// 1. Inner data table -- the actual payload
table Xxx {
    field_a: SomeType (id: 0);
    field_b: AnotherType (id: 1);
}

// 2. Tracked wrapper -- used by the in-memory tracker API.
//    data is null when the tracked entity is inactive.
table XxxTracked {
    data: Xxx (id: 0);
}

// 3. Record wrapper -- used as the MCAP recording root type.
//    Adds a DeviceDataTimestamp alongside the payload.
table XxxRecord {
    data: Xxx (id: 0);
    timestamp: DeviceDataTimestamp (id: 1);
}

root_type XxxRecord;

  • Inner data table (e.g. HeadPose, HandPose, ControllerSnapshot) – contains the device-specific fields. All fields are present when the parent wrapper’s data pointer is non-null.

  • Tracked wrapper (e.g. HeadPoseTracked) – wraps the inner data in an optional data field. The in-memory get_*() accessors return a reference to this wrapper. When data is nullptr (C++) or None (Python), the device is inactive or no sample has arrived yet.

  • Record wrapper (e.g. HeadPoseRecord) – wraps the inner data plus a DeviceDataTimestamp. This is the root_type written to MCAP channels by the recorder via serialize_all().

Shared Types#

DeviceDataTimestamp (src/core/schema/fbs/timestamp.fbs)

All timestamp fields are int64 nanoseconds.

Field

Description

available_time_local_common_clock

System monotonic time when the sample became available to the recording system. Useful for measuring pipeline latency.

sample_time_local_common_clock

System monotonic time when the sample was captured. Enables cross-device synchronization (values from different devices share the same clock domain).

sample_time_raw_device_clock

Timestamp from the device’s own clock. Values from different devices are not directly comparable.

Pose (src/core/schema/fbs/pose.fbs)

struct Point      { x: float; y: float; z: float; }
struct Quaternion  { x: float; y: float; z: float; w: float; }
struct Pose {
  position: Point;       // meters
  orientation: Quaternion;
}

Tracker Reference#

HeadTracker#

Tracks the HMD head pose via the OpenXR view space.

HandTracker#

Tracks articulated hand joints (26 joints per hand, following the OpenXR XrHandJointEXT ordering) using the XR_EXT_hand_tracking extension.

ControllerTracker#

Tracks both left and right controllers – grip and aim poses, plus button and axis inputs. Uses standard OpenXR action bindings.

FullBodyTrackerPico#

Tracks 24 body joints on PICO devices using the XR_BD_body_tracking extension.

FrameMetadataTrackerOak#

Multi-channel tracker for per-frame metadata from OAK camera streams. Uses the SchemaTracker utility internally.

Generic3AxisPedalTracker#

Reads foot pedal axis values pushed by a device plugin through OpenXR tensor collections. Uses the SchemaTracker utility internally.

Note

The Python method is named get_pedal_data() (instead of the C++ get_data()).

Usage Examples#

For end-to-end usage patterns combining trackers with a DeviceIOSession, see:

For higher-level usage with the teleop session manager and retargeting, see: