benchmarks#

Classes

BenchmarkResult

Functions

run_aiperf_benchmark

Compatibility wrapper for callers that request one concurrency.

run_aiperf_sweep

Run a serving sweep while preserving offline and remote-code policy by default.

write_aiperf_report

class BenchmarkResult#

Bases: StrictModel

architecture_id: str#
cache_identity: str#
checkpoint_dir: str#
command: tuple[str, ...]#
completed_at: datetime#
concurrency: int#
engine: Literal['aiperf']#
failures: int#
gpu_count: int#
metrics: dict[str, float]#
model_config = {'extra': 'forbid', 'frozen': True}#

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

profile_id: str#
raw_artifacts: dict[str, str]#
solution_id: str#
started_at: datetime#
topology: dict[str, Any]#
topology_id: str#
workload: dict[str, Any]#
workload_id: str#
run_aiperf_benchmark(checkpoint_dir, *, artifact_dir, concurrency, input_tokens, output_tokens, gpu_ids, topology, request_count, solution_id='unknown', profile_id='unknown', topology_id=None, **kwargs)#

Compatibility wrapper for callers that request one concurrency.

Parameters:

  • checkpoint_dir (str | Path)

  • artifact_dir (str | Path)

  • concurrency (int)

  • input_tokens (int)

  • output_tokens (int)

  • gpu_ids (str)

  • topology (dict[str, Any])

  • request_count (int)

  • solution_id (str)

  • profile_id (str)

  • topology_id (str | None)

Return type:

BenchmarkResult

run_aiperf_sweep(checkpoint_dir, *, artifact_dir, concurrencies, input_tokens, output_tokens, gpu_ids, topology, request_counts=None, solution_id='unknown', profile_id='unknown', topology_id=None, executable='aiperf', endpoint_type='chat', extra_inputs=None, use_server_token_count=True, seed=42, readiness_timeout=1200, benchmark_timeout=600, gpu_telemetry='pynvml', trust_remote_code=False, allow_aiperf_v011_online_tokenizer_resolution=False)#

Run a serving sweep while preserving offline and remote-code policy by default.

Parameters:

  • checkpoint_dir (str | Path)

  • artifact_dir (str | Path)

  • concurrencies (Iterable[int])

  • input_tokens (int)

  • output_tokens (int)

  • gpu_ids (str)

  • topology (dict[str, Any])

  • request_counts (dict[int, int] | None)

  • solution_id (str)

  • profile_id (str)

  • topology_id (str | None)

  • executable (str | Path)

  • endpoint_type (str)

  • extra_inputs (dict[str, Any] | None)

  • use_server_token_count (bool)

  • seed (int)

  • readiness_timeout (float)

  • benchmark_timeout (float)

  • gpu_telemetry (str | None)

  • trust_remote_code (bool)

  • allow_aiperf_v011_online_tokenizer_resolution (bool)

Return type:

list[BenchmarkResult]

write_aiperf_report(results, output_dir)#
Parameters:

  • results (list[Any])

  • output_dir (str | Path)

Return type:

dict[str, str]