SimReady Benchmark Overview#

Benchmarking goes beyond schema validation. It opens a SimReady asset in the target engine, NVIDIA Kit, or NVIDIA Isaac Sim, exercises it, and records what happens. Schema validation, through the SimReady Validation Workflow, checks an asset against the SimReady requirements for a profile without running the engine. Benchmarking catches behavior that schema validation cannot detect before the asset reaches a production pipeline.

The Tool and the Tests#

Benchmarking is driven by the simready-benchmark tool. The bundled FET test families are part of the SimReady Foundation and target NVIDIA Kit and NVIDIA Isaac Sim. The simready-benchmark command discovers them from the foundation automatically, so you do not need to install a separate test suite.

Test Families#

Each asset runs only the tests that its features and profiles call for.

The bundled suite groups tests into families by feature, as shown in the following table:

Family

Validates

FET001

Visual presence, normals, back-face culling, and lighting.

FET003

Physics ground and slope drop, and resting stability.

FET004

Multi-body and joint discovery and movement.

FET005

Grasp and lift, where a gripper grasps the asset and raises it.

FET022

Driven joints: range, velocity, mimic behavior, coordination, and inverse kinematics (IK).

Relationship to Features and Profiles#

Benchmarks are selected from the features and profiles an asset declares. This is the same vocabulary used throughout the SimReady specification. A profile bundles features, and each feature maps to the benchmarks that confirm it. Refer to Pipeline and Stages for how those declarations become a test plan.

Next Steps#