SimReady Benchmark Overview#

Benchmarking goes beyond schema validation. It opens a SimReady asset in the target engine, NVIDIA Kit, or NVIDIA Isaac Sim, exercises it, and records what happens. Schema validation, through the SimReady Validation Workflow, checks an asset against the SimReady requirements for a profile without running the engine. Benchmarking catches behavior that schema validation cannot detect before the asset reaches a production pipeline.

The Tool and the Tests#

Benchmarking is driven by the simready-benchmark tool. The bundled FET test families are part of the SimReady Foundation and target NVIDIA Kit and NVIDIA Isaac Sim. An installed Foundation tier advertises its catalogs and bundled runtime-test package through the simready.tier entry point. During source development, --foundations-path selects a Foundation checkout; --tests-path is reserved for adding an independent unpublished test package. Running Tests shows each workflow.

Test Families#

Each asset runs only the tests that its features and profiles call for.

The bundled suite groups tests into families by feature, as shown in the following table:

Family labels such as FET001 are compact reporting groups and CLI filter prefixes, not feature manifest IDs. Runtime results are attributed through the canonical IDs declared by each test, such as FET_001_STANDARD. Some families map to multiple runtime variants, so tools must read the test declarations rather than derive a manifest name from the family label.

Family

Validates

FET001

Visual presence, normals, back-face culling, and lighting.

FET003

Ground drop with settling and slope drop for Standard, PhysX, and Newton rigid bodies.

FET004

Multi-body and joint discovery and movement.

FET005

Grasp and lift, where a gripper grasps the asset and raises it.

FET011

Semantic-label behavior in the RTX segmentation pipeline.

FET022

Driven joints: range, velocity, mimic behavior, coordination, and inverse kinematics (IK).

FET028

Gripper close, lift, shake, and drop.

Only enabled tests registered by the bundled tier suite are documented here. Use simready-benchmark --list-tests to inspect the tests available in the current environment, including tests contributed by other installed tiers or independent test packs.

Relationship to Features and Profiles#

Benchmarks are selected from the features and profiles an asset declares. This is the same vocabulary used throughout the SimReady specification. A profile bundles features, and each feature maps to the benchmarks that confirm it. Refer to Pipeline and Stages for how those declarations become a test plan.

Next Steps#