benchmarks

Functions

artifact_sha256

Hash one retained benchmark artifact.

benchmark_result_fingerprint

Return an integrity fingerprint for one self-contained result row.

checkpoint_identity

Describe and content-address a serialized checkpoint.

executable_identity

Content-address a resolved executable and its supported distribution.

hardware_identity

Return host and visible-accelerator identity without requiring a GPU locally.

run_aiperf_benchmark

Compatibility wrapper for callers that request one concurrency.

run_aiperf_sweep

Run a serving sweep while preserving offline and remote-code policy by default.

software_identity

Return the runtime versions that can affect serving measurements.

write_aiperf_report

ModeloptConfig BenchmarkResult

Bases: StrictModel

Show default config as JSON
Default config (JSON):

{
   "architecture_id": null,
   "checkpoint_dir": null,
   "solution_id": "unknown",
   "profile_id": "unknown",
   "topology_id": "unknown",
   "workload_id": "unknown",
   "repetition": 0,
   "gpu_count": 1,
   "cache_identity": "unknown",
   "engine": "aiperf",
   "topology": null,
   "workload": null,
   "concurrency": null,
   "metrics": null,
   "measurement_contract": null,
   "checkpoint_identity": null,
   "hardware_identity": null,
   "software_identity": null,
   "result_fingerprint": "unknown",
   "failures": 0,
   "raw_artifacts": null,
   "raw_artifact_sha256": null,
   "command": null,
   "started_at": null,
   "completed_at": null
}

field architecture_id: str [Required]
field cache_identity: str
field checkpoint_dir: str [Required]
field checkpoint_identity: dict[str, Any] [Optional]
field command: tuple[str, ...] [Required]
field completed_at: datetime [Optional]
field concurrency: int [Required]
Constraints:

  • ge = 1

field engine: Literal['aiperf']
field failures: int
Constraints:

  • ge = 0

field gpu_count: int
Constraints:

  • ge = 1

field hardware_identity: dict[str, Any] [Optional]
field measurement_contract: dict[str, Any] [Optional]
field metrics: dict[str, float] [Required]
field profile_id: str
field raw_artifact_sha256: dict[str, str] [Optional]
field raw_artifacts: dict[str, str] [Required]
field repetition: int
Constraints:

  • ge = 0

field result_fingerprint: str
field software_identity: dict[str, Any] [Optional]
field solution_id: str
field started_at: datetime [Required]
field topology: dict[str, Any] [Required]
field topology_id: str
field workload: dict[str, Any] [Required]
field workload_id: str
artifact_sha256(path)

Hash one retained benchmark artifact.

Parameters:

path (str | Path)

Return type:

str

benchmark_result_fingerprint(payload)

Return an integrity fingerprint for one self-contained result row.

Parameters:

payload (dict[str, Any])

Return type:

str

checkpoint_identity(checkpoint_dir)

Describe and content-address a serialized checkpoint.

Parameters:

checkpoint_dir (str | Path)

Return type:

dict[str, Any]

executable_identity(executable, *, distribution_name='aiperf')

Content-address a resolved executable and its supported distribution.

Parameters:

  • executable (str | Path)

  • distribution_name (str | None)

Return type:

dict[str, Any]

hardware_identity(gpu_ids)

Return host and visible-accelerator identity without requiring a GPU locally.

Parameters:

gpu_ids (str)

Return type:

dict[str, Any]

run_aiperf_benchmark(checkpoint_dir, *, artifact_dir, concurrency, input_tokens, output_tokens, gpu_ids, topology, request_count, solution_id='unknown', profile_id='unknown', topology_id=None, **kwargs)

Compatibility wrapper for callers that request one concurrency.

Parameters:

  • checkpoint_dir (str | Path)

  • artifact_dir (str | Path)

  • concurrency (int)

  • input_tokens (int)

  • output_tokens (int)

  • gpu_ids (str)

  • topology (dict[str, Any])

  • request_count (int)

  • solution_id (str)

  • profile_id (str)

  • topology_id (str | None)

Return type:

BenchmarkResult

run_aiperf_sweep(checkpoint_dir, *, artifact_dir, concurrencies, input_tokens, output_tokens, gpu_ids, topology, request_counts=None, warmup_request_count=0, warmup_seed=None, repetitions=1, collect_peak_gpu_memory=False, solution_id='unknown', profile_id='unknown', topology_id=None, executable='aiperf', endpoint_type='chat', extra_inputs=None, use_server_token_count=True, seed=42, readiness_timeout=1200, benchmark_timeout=600, gpu_telemetry='pynvml', image_batch_sizes=None, image_width_mean=0, image_height_mean=0, trust_remote_code=False, allow_aiperf_v011_online_tokenizer_resolution=False)

Run a serving sweep while preserving offline and remote-code policy by default.

Parameters:

  • checkpoint_dir (str | Path)

  • artifact_dir (str | Path)

  • concurrencies (Iterable[int])

  • input_tokens (int)

  • output_tokens (int)

  • gpu_ids (str)

  • topology (dict[str, Any])

  • request_counts (dict[int, int] | None)

  • warmup_request_count (int)

  • warmup_seed (int | None)

  • repetitions (int)

  • collect_peak_gpu_memory (bool)

  • solution_id (str)

  • profile_id (str)

  • topology_id (str | None)

  • executable (str | Path)

  • endpoint_type (str)

  • extra_inputs (dict[str, Any] | None)

  • use_server_token_count (bool)

  • seed (int)

  • readiness_timeout (float)

  • benchmark_timeout (float)

  • gpu_telemetry (str | None)

  • image_batch_sizes (Iterable[int] | None)

  • image_width_mean (int)

  • image_height_mean (int)

  • trust_remote_code (bool)

  • allow_aiperf_v011_online_tokenizer_resolution (bool)

Return type:

list[BenchmarkResult]

software_identity()

Return the runtime versions that can affect serving measurements.

Return type:

dict[str, Any]

write_aiperf_report(results, output_dir)
Parameters:
Return type:

dict[str, str]