Add a New Device#

To add a new device that streams typed data over the OpenXR runtime, follow these four steps. The reference implementation is the generic 3-axis foot pedal:

Component

Code Location

Data Schema

src/core/schema/fbs/pedals.fbs

Device Plugin

src/plugins/generic_3axis_pedal

Tracker facade (Generic3AxisPedalTracker)

src/core/deviceio_trackers/cpp/generic_3axis_pedal_tracker.cpp and src/core/deviceio_trackers/cpp/inc/deviceio/generic_3axis_pedal_tracker.hpp

Live backend (LiveGeneric3AxisPedalTrackerImpl)

src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.cpp and src/core/live_trackers/cpp/live_generic_3axis_pedal_tracker_impl.hpp

SchemaTracker / SampleResult

src/core/live_trackers/cpp/inc/live_trackers/schema_tracker.hpp

Debug printer

examples/schemaio/pedal_printer.cpp

Step 1: Define the data schema#

Define a FlatBuffer schema (.fbs) under src/core/schema/fbs. The schema drives both serialization in the plugin and deserialization in the tracker; pusher and reader must agree on the same schema ahead of time (the schema is not sent over the wire).

Reference schema: src/core/schema/fbs/pedals.fbs

include "point.fbs";
include "timestamp.fbs";

namespace core;

table Generic3AxisPedalOutput {
  left_pedal: float (id: 0);
  right_pedal: float (id: 1);
  rudder: float (id: 2);
}

table Generic3AxisPedalOutputRecord {
  data: Generic3AxisPedalOutput (id: 0);
  timestamp: DeviceDataTimestamp (id: 1);
}

  • Output table — The primary payload type (e.g. Generic3AxisPedalOutput) with the device fields. This is what the plugin serializes and pushes.

  • Tracked wrapper — A table that wraps the output in an optional data field (e.g. Generic3AxisPedalOutputTracked). Used by the in-memory tracker API so that data can be null when no sample is available.

  • Record wrapper — A table that wraps the output plus DeviceDataTimestamp (e.g. Generic3AxisPedalOutputRecord). This is the root type written to MCAP channels by the recorder; trackers serialize into this type in serialize_all().

  • root_type — Set to the Record type (e.g. root_type Generic3AxisPedalOutputRecord;).

Include timestamp.fbs for DeviceDataTimestamp; include other shared types (e.g. point.fbs) as needed. After adding or changing a schema, rebuild so that the C++ and Python generated code (e.g. pedals_generated.h, pedals_bfbs_generated.h) is updated.

Step 2: Implement a device plugin#

The plugin runs in a separate process (or as part of a host app), reads hardware, and pushes serialized FlatBuffer data via OpenXR using the SchemaPusher from the pusherio library. Reuse the same pattern as the example apps in examples/schemaio/ (e.g. pedal_pusher which uses SchemaPusher).

  • OpenXR session — Create an OpenXRSession with extensions from SchemaPusher::get_required_extensions() (includes XR_NVX1_push_tensor and XR_NVX1_tensor_data).

  • SchemaPusher — Construct a SchemaPusher with OpenXRSessionHandles and a SchemaPusherConfig: collection_id, max_flatbuffer_size, tensor_identifier, localized_name, and optionally app_name. The collection_id and tensor_identifier must match what the tracker uses.

  • Push loop — In your update loop, fill the schema’s native type (e.g. Generic3AxisPedalOutputT), serialize it with FlatBufferBuilder and the generated Pack(), then call pusher_.push_buffer(ptr, size, sample_time_local_common_clock_ns, sample_time_raw_device_clock_ns). Use a monotonic clock (e.g. core::os_monotonic_now_ns()) for the local common clock; use the device’s own clock for the raw device clock if available.

Reference implementation: src/plugins/generic_3axis_pedal. The plugin holds a core::SchemaPusher member, opens the Linux joystick device, maps axes to Generic3AxisPedalOutputT, and calls push_current_state() from update(). See generic_3axis_pedal_plugin.hpp and generic_3axis_pedal_plugin.cpp.

Step 3: Implement a tracker#

The tracker runs inside a consumer process (e.g. Teleop pipeline or a small reader app). It implements the ITracker interface (tracker facade in deviceio_trackers); the live backend in live_trackers composes SchemaTracker to read raw tensor samples from OpenXR. Implement a concrete tracker class (e.g. Generic3AxisPedalTracker) that:

  • Extends ITracker — Override get_name(), get_schema_name(), get_schema_text(), and get_record_channels(). For OpenXR, add a static required_extensions() on your live ITrackerImpl and register the tracker type in LiveDeviceIOFactory::get_required_extensions (tensor/schema readers usually forward SchemaTrackerBase::get_required_extensions()). Callers use DeviceIOSession::get_required_extensions(trackers). Return the Record type name and binary schema for MCAP; return at least one channel name.

  • Holds user configuration — Same logical inputs as the pusher (e.g. collection_id, max_flatbuffer_size). The live ITrackerImpl builds the internal tensor settings (SchemaTrackerConfig) so they match the plugin.

  • Factory registration — Register your tracker in the live factory dispatch table (see LiveDeviceIOFactory). The factory constructs an ITrackerImpl that holds a SchemaTracker, builds a SchemaTrackerConfig from the tracker’s stored configuration, and implements update(XrTime) and serialize_all(channel_index, callback).

In the Impl:

  • update() — Call m_schema_reader.read_all_samples(pending_records). If the collection is not present, clear the tracked state (e.g. set m_tracked.data = nullptr). Otherwise, deserialize the latest sample (or all samples) into your tracked type and keep the last one for get_data().

  • serialize_all() — For each sample in the pending batch, deserialize, build the Record FlatBuffer (output table + DeviceDataTimestamp), and invoke the callback with (log_time_ns, buffer_ptr, size). The buffer is only valid during the callback. If the device disappeared and there are no samples, you may emit one record with null data and the update-tick timestamp so the MCAP stream marks absence.

Reference implementation — split across facade and live backend:

Step 4: Implement a simple C++ printer (optional)#

A minimal reader app verifies the full path: plugin (or pusher) pushes; printer discovers the collection and prints samples. Pattern (see examples/schemaio/pedal_printer.cpp):

  1. Create the tracker (e.g. std::make_shared<Generic3AxisPedalTracker>(collection_id, max_flatbuffer_size)).

  2. Get required extensions with DeviceIOSession::get_required_extensions(trackers) and create an OpenXRSession.

  3. Create a DeviceIOSession with DeviceIOSession::run(trackers, oxr_session->get_handles()).

  4. Loop: call session->update(), then read tracker->get_data(*session). If tracked.data is non-null, use the latest sample; otherwise sleep briefly and repeat.

Use the same collection_id (and optionally tensor_identifier) as the plugin. See Schema IO example: build and run above for building and running pedal_pusher and pedal_printer.

Schema IO example: build and run#

The Schema IO example is a simpler example that demonstrates pushing and reading serialized FlatBuffer data via the OpenXR runtime using the Generic Tensor Collection interface. It provides two binaries: pedal_pusher (serializes and pushes Generic3AxisPedalOutput using SchemaPusher) and pedal_printer (reads via Generic3AxisPedalTracker and DeviceIOSession). Both use the XR_NVX1_push_tensor and XR_NVX1_tensor_data extensions. Pusher and reader agree on the schema (Generic3AxisPedalOutput from pedals.fbs), so the schema is not sent over the wire.

Build (from the project root, with examples enabled):

cmake -B build -DBUILD_EXAMPLES=ON
cmake --build build --parallel
cmake --install build

Run pusher and printer in separate terminals:

# Terminal 1: Start printer
./install/examples/schemaio/pedal_printer

# Terminal 2: Start pusher
./install/examples/schemaio/pedal_pusher

The printer discovers the tensor collection created by the pusher and prints received samples. Both exit after 100 samples, or press Ctrl+C to exit early.

Components

  • SchemaPusher (pusherio library) — Pushes serialized FlatBuffer data via OpenXR tensor extensions: takes externally-provided OpenXR session handles, creates a tensor collection with the configured identifier, provides push_buffer() for raw serialized data. Use composition to create typed wrappers (e.g. Generic3AxisPedalPusher in examples/schemaio/pedal_pusher.cpp).

  • SchemaTracker (live_trackers) — Helper for reading FlatBuffer schema data via OpenXR tensor collections: discovers collections by identifier, exposes read_all_samples() into SampleResult values. Live tracker implementations (e.g. LiveGeneric3AxisPedalTrackerImpl) compose a SchemaTracker and implement ITrackerImpl::update() / serialize_all().

  • Generic3AxisPedalTracker (tracker facade in deviceio_trackers) — Concrete ITracker for Generic3AxisPedalOutput: holds configuration and get_data(session) returning Generic3AxisPedalOutputTrackedT via the session’s IGeneric3AxisPedalTrackerImpl.

  • DeviceIOSession — Session manager: collects required OpenXR extensions from registered trackers, creates tracker implementations with session handles, and calls update() on all trackers during the update loop.