runtime_vllm

vLLM Runtime Benchmark Integration for ModelOpt NAS Subblocks.

This module provides the integration logic to empirically benchmark subblock runtime statistics within transformer architectures using the vLLM latency benchmark. Each invocation is launched in a dedicated subprocess so that GPU memory and CUDA state are fully reclaimed when the subprocess exits, allowing many sequential benchmarks to run in a single Python session without leaking.

Usage:

• Call run_vllm_latency_benchmark with a model path and a RuntimeConfig instance to run a latency benchmark and return the average latency for the configuration (in milliseconds).

Classes

RuntimeMeasurement

Additive phase timing carried through repeated-block differencing.

Functions

run_vllm_latency_benchmark

Measure combined and prefill-only latency for one serialized model.

class RuntimeMeasurement

Bases: object

Additive phase timing carried through repeated-block differencing.

__init__(total_ms, prefill_ms)

Parameters:

• total_ms (float)

• prefill_ms (float)

Return type:

None

property decode_ms: float

decode_ms_per_token(generation_seq_len)

Parameters:

generation_seq_len (int)

Return type:

float

classmethod from_dict(value)

Parameters:

value (dict[str, float])

Return type:

RuntimeMeasurement

classmethod mean(values)

Return type:

RuntimeMeasurement

prefill_ms: float

to_dict()

Return type:

dict[str, float]

total_ms: float

classmethod zero()

Return type:

RuntimeMeasurement

run_vllm_latency_benchmark(model_path, runtime_config, gpu_id=None, cache_dir=None)

Measure combined and prefill-only latency for one serialized model.

The prefill workload generates one token. Decode latency is intentionally derived later as combined - prefill so repeated-block differencing is performed component-wise before the phase split is exposed to consumers.

Parameters:

• model_path (Path)

• runtime_config (RuntimeConfig)

• gpu_id (str | int | None)

• cache_dir (Path | None)

Return type:

RuntimeMeasurement