Authoring and Execution Deep Dive

This guide explains how runtime tests are authored and executed by job_runner.py.

Components

• Test definitions (TOML)

• Search functions (Python) returning assets

• Runner info (TOML) produced and maintained by the user/CI (batch_maker reads it)

• RuntimeTest Python classes (your test logic)

• Job JSON (generated by batch_maker; do not hand-edit or generate manually)

Test definitions (TOML)

Sections:

• [TestInfo]

  • name (string)

  • description (string)

  • version (string)

  • search_function (string)

  • optional: disabled (bool)

  • optional: exclude_assets (comma-separated string)

  • optional: exclude_assets_list (JSON file path)

  • optional: features ([[TestInfo.features]] array of tables)

    • id (string)

    • version (string)

• [RunnerTags]

  • runner_config (array of runner names defined in runner info)

  • optional per-runner overrides are supported under tables named after runner names

  • optional: test_file (Python file path for script-based modes)

  • optional: test_class (RuntimeTest subclass name)

• [TestConfig]

  • free-form test-specific configuration (dict)

Example:

[TestInfo]
name = "Ground Drop"
description = "Simple physics test"
version = "1.0.0"
search_function = "feature_rbd_physics_physx_v_0_1_0"

[[TestInfo.features]]
id = "FET003_BASE_PHYSX"
version = "0.1.0"

[RunnerTags]
runner_config = ["kit_runner_windows_gpu"]
test_file = "test_definitions/physics_tests/ground_drop.py"
test_class = "GroundDrop"

[TestConfig]
capture_prefix = "ground_drop_"

Search functions

• Reside under search_functions directory.

• File and function name must match.

• Signature: def my_search(context) -> list[AssetData]

• Use the provided AssetSearchContext and filters to return assets.

Run modes and runner selection

• Run mode is deduced from RunnerTags/runner config in runners_info.toml.

• Script-required modes require test_file (and for per-asset script modes, test_class).

• Engine modes must not set test_file.

• Platforms filter is space-separated list via batch_maker --platforms windows linux.

Note: runners_info.toml is maintained by the user/CI. Required fields include: engine (set to "kit"), platform, project_root, executable_path, number_of_runners. Optional: output_dir, docker_image, hardware_gpu, runner_config_file.

Writing a RuntimeTest

Base class: core.runtime_test.RuntimeTest

• Synchronous override: def run_test(self) -> None

• Asynchronous override: async def run_async(self) -> None

• self.context is set with job, asset, and result helpers.

Minimal pattern:

from core.runtime_test import RuntimeTest

class MyTest(RuntimeTest):
    def run_test(self) -> None:
        assert self.context is not None
        # Implement test logic using self.context
        # Optionally add artifacts/metrics to result
        self.context.test_result.add_metric("example", 1)

See nv_core/testing_tools/test_definitions/kit_test/minimal_app.py for a more complete example (omit presentation extras if not needed).

Recording results with TestExecutionResult

common.test_execution_result.TestExecutionResult is available via self.context.test_result:

• add_metric(name, value)

• add_screenshot(path)

• add_video(path)

• add_usd_file(path)

• add_log(message)

• The framework tracks passed, execution_time, and error_message.

The base class handles timing and finalization. Callers typically add artifacts and metrics; exceptions or manual failure will mark the test failed.

Execution flow

1. batch_maker discovers tests/assets and writes jobs under _testing/batch_jobs/.

2. job_runner loads a job file, prepares output dirs under _testing/job_outputs/, sets environment, and invokes the engine/script runner.

3. Each asset is executed; RuntimeTest writes XML and artifacts into the asset folder.

4. report_generator parses {output}/batch_jobs and {output}/job_outputs, building JSON and HTML summaries.

Tips

• Use --keep-outputs appropriately (see overview).

• For distributed jobs, select the intended job file and call job_runner.py directly.

• Validate search functions and test definitions early; batch_maker will emit actionable warnings.

Pre-check-in workflow (recommended)

• Narrow test scope to changed assets:

# Build a JSON array of changed asset relative paths
["sample_content/common_assets/ModelA.usd", "sample_content/common_assets/ModelB.usd"]

# Use it with batch_maker
workspace runtime_tests batch_maker --modified-assets-list D:/simready/modified_assets.json
workspace runtime_tests job_runner --enable-zip
workspace runtime_tests report_generator --output-dir D:/simready/_testing

• Review {project_root}/_testing/index.html and address failures before committing.

Additional examples

• Generate jobs for both platforms but only run locally on current platform:

workspace runtime_tests batch_maker --project-root D:/simready --platforms windows linux
workspace runtime_tests job_runner

• Run a specific runner instance job file (distributed):

python job_runner/job_runner.py D:/simready/_testing/batch_jobs/kit_runner_windows_gpu_001_windows.json --enable-zip --timeout 1800

Troubleshooting (quick)

• Missing search library: ensure --search-library-path (CLI) or run via workspace runtime_tests (auto-discovery).

• No jobs generated: verify search function names, filters, runner names exist in runner info, and features match.

• No XML results: engine may have failed; inspect job_outputs folders, engine logs, and consider enabling --enable-zip.

• Do not hand-edit job JSON files; always use batch_maker outputs.