trtllm-bench#

trtllm-bench is a comprehensive benchmarking tool for TensorRT LLM engines. It provides subcommands for different benchmarking scenarios.

Note

Non-breaking: --config <file.yaml> is the preferred flag for passing a YAML configuration file. Existing workflows using --extra_llm_api_options <file.yaml> continue to work; it is an equivalent alias.

Syntax#

trtllm-bench#

Usage

trtllm-bench [OPTIONS] COMMAND [ARGS]...

Options

-m, --model <model>#: Required The Huggingface name of the model to benchmark. Also used as the tokenizer source for the prepare-dataset subcommand.

--model_path <model_path>#: Path to a Huggingface checkpoint directory for loading model components.

-w, --workspace <workspace>#: The directory to store benchmarking intermediate files.

--log_level <log_level>#

The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--revision <revision>#: The revision to use for the HuggingFace model (branch name, tag name, or commit id).

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

throughput#

Run a throughput test on a TRT-LLM engine.

Usage

trtllm-bench throughput [OPTIONS]

Options

--engine_dir <engine_dir>#: Path to a serialized TRT-LLM engine.

--backend <backend>#

The backend to use for benchmark. Default is pytorch backend.

Options:: pytorch | _autodeploy

--custom_module_dirs <custom_module_dirs>#: Paths to custom module directories to import.

--config, --extra_llm_api_options <extra_llm_api_options>#: Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file. Can be specified as either –config or –extra_llm_api_options.

--sampler_options <sampler_options>#: Path to a YAML file that sets sampler options.

--max_batch_size <max_batch_size>#: Maximum runtime batch size to run the engine with.

--max_num_tokens <max_num_tokens>#: Maximum runtime tokens that an engine can accept.

--max_seq_len <max_seq_len>#: Maximum sequence length.

--beam_width <beam_width>#: Number of search beams.

--kv_cache_free_gpu_mem_fraction <kv_cache_free_gpu_mem_fraction>#: The percentage of memory to use for KV Cache after model load.

--dataset <dataset>#: Pass in a dataset file for parsing instead of stdin.

--no_skip_tokenizer_init#: Do not skip tokenizer initialization when loading the model.

--custom_tokenizer <custom_tokenizer>#: Custom tokenizer alias (e.g., ‘deepseek_v32’) or fully-qualified ‘module.path.ClassName’ for models whose HF tokenizer is incompatible with AutoTokenizer.

--eos_id <eos_id>#: Set the end-of-sequence token for the benchmark. Set to -1 to disable EOS.

--modality <modality>#

Modality of the multimodal requests.

Options:: image | video

--image_data_format <image_data_format>#

Format of the image data for multimodal models.

Options:: pt | pil

--data_device <data_device>#

Device to load the multimodal data on.

Options:: cuda | cpu

--max_input_len <max_input_len>#: Maximum input sequence length to use for multimodal models. This is used only when –modality is specified since the actual number of vision tokens is unknown before the model is run.

--num_requests <num_requests>#: Number of requests to cap benchmark run at. If not specified or set to 0, it will be the length of dataset.

--duration <duration>#: Maximum run time in seconds. Benchmark stops at whichever limit is hit first (num_requests or duration). Requires –concurrency. Requests dropped at the deadline are excluded from the report, so the statistics cover the requests that completed rather than the whole dataset.

--warmup <warmup>#: Number of requests warm up benchmark.

--target_input_len <target_input_len>#: Target (average) input length for tuning heuristics.

--target_output_len <target_output_len>#: Target (average) sequence length for tuning heuristics.

--tp <tp>#: tensor parallelism size

--pp <pp>#: pipeline parallelism size

--ep <ep>#: expert parallelism size

--cluster_size <cluster_size>#: expert cluster parallelism size

--concurrency <concurrency>#: Desired concurrency rate (number of requests processing at the same time), <=0 for no concurrency limit.

--streaming#: Enable streaming mode for requests.

--report_json <report_json>#: Path where report is written to.

--iteration_log <iteration_log>#: Path where iteration logging is written to.

--output_json <output_json>#: Path where output should be written to.

--request_json <request_json>#: Path where per request information is written to.

--enable_chunked_context, --disable_chunked_context#: Enable/disable chunking in prefill stage for enhanced throughput benchmark.

--scheduler_policy <scheduler_policy>#

KV cache scheduler policy: guaranteed_no_evict prevents request eviction, max_utilization optimizes for throughput.

Options:: guaranteed_no_evict | max_utilization

latency#

Run a latency test on a TRT-LLM engine.

Usage

trtllm-bench latency [OPTIONS]

Options

--engine_dir <engine_dir>#: Path to a serialized TRT-LLM engine.

--config, --extra_llm_api_options <extra_llm_api_options>#: Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file. Can be specified as either –config or –extra_llm_api_options.

--backend <backend>#

The backend to use for benchmark. Default is pytorch backend.

Options:: pytorch | _autodeploy

--kv_cache_free_gpu_mem_fraction <kv_cache_free_gpu_mem_fraction>#: The percentage of memory to use for KV Cache after model load.

--max_seq_len <max_seq_len>#: Maximum sequence length.

--dataset <dataset>#: Pass in a dataset file for parsing instead of stdin.

--modality <modality>#

Modality of the multimodal requests.

Options:: image | video

--max_input_len <max_input_len>#: Maximum input sequence length to use for multimodal models. This is used only when –modality is specified since the actual number of vision tokens is unknown before the model is run.

--custom_tokenizer <custom_tokenizer>#: Custom tokenizer alias (e.g., ‘deepseek_v32’) or fully-qualified ‘module.path.ClassName’ for models whose HF tokenizer is incompatible with AutoTokenizer.

--num_requests <num_requests>#: Number of requests to cap benchmark run at. Minimum between value andlength of dataset.

--duration <duration>#: Maximum run time in seconds. Benchmark stops at whichever limit is hit first (num_requests or duration). Requests dropped at the deadline are excluded from the report, so the statistics cover the requests that completed rather than the whole dataset.

--warmup <warmup>#: Number of requests warm up benchmark.

--tp <tp>#: tensor parallelism size

--pp <pp>#: pipeline parallelism size

--ep <ep>#: expert parallelism size

--beam_width <beam_width>#: Number of search beams.

--sampler_options <sampler_options>#: Path to a YAML file that sets sampler options.

--concurrency <concurrency>#: Desired concurrency rate (number of requests processing at the same time), <=0 for no concurrency limit.

--medusa_choices <medusa_choices>#: Path to a YAML file that defines the Medusa tree.

--report_json <report_json>#: Path where report should be written to.

--iteration_log <iteration_log>#: Path where iteration logging is written to.

prepare-dataset#

Prepare dataset for benchmarking with trtllm-bench.

Usage

trtllm-bench prepare-dataset [OPTIONS] COMMAND [ARGS]...

Options

--output <output>#: Output json filename.

--stdout#: Print the dataset to stdout with a JSON entry on each line instead of writing a file.

--random-seed <random_seed>#: random seed for token_ids

--task-id <task_id>#: LoRA task id

--rand-task-id <rand_task_id>#: Random LoRA Tasks

--lora-dir <lora_dir>#: Directory containing LoRA adapters

--log-level <log_level>#

Logging level.

Options:: info | debug

--trust-remote-code#: Trust remote code.

Environment variables

TRUST_REMOTE_CODE: Provide a default for --trust-remote-code

real-dataset#

Prepare dataset from real dataset.

Supports three input modes based on the shape of the data at –dataset-input-key:

Single-turn text (default): The value is a string concatenated with the prompt from –dataset-prompt-key / –dataset-prompt.
Multi-turn conversation: The value is a list of strings (e.g. MT-Bench turns field). Each list element is one conversation turn. No –dataset-prompt-key / –dataset-prompt is required.
Multimodal: Detected when the row contains image / video keys.

Usage

trtllm-bench prepare-dataset real-dataset [OPTIONS]

Options

--dataset-name <dataset_name>#: Required Dataset name in HuggingFace.

--dataset-config-name <dataset_config_name>#: Dataset config name in HuggingFace (if exists).

--dataset-split <dataset_split>#: Required Split of the dataset to use.

--dataset-input-key <dataset_input_key>#: The dataset dictionary key for input.

--dataset-image-key <dataset_image_key>#: The dataset dictionary key for images.

--dataset-prompt-key <dataset_prompt_key>#: The dataset dictionary key for prompt (if exists).

--dataset-prompt <dataset_prompt>#: The prompt string when there is no prompt key for the dataset.

--dataset-output-key <dataset_output_key>#: The dataset dictionary key for output (if exists).

--num-requests <num_requests>#: Number of requests to be generated. Will be capped to min(dataset.num_rows, num_requests).

--max-input-len <max_input_len>#: Maximum input sequence length for a given request. This will be used to filter out the requests with long input sequence length. Default will include all the requests.

--output-len-dist <output_len_dist>#: Output length distribution. Default will be the length of the golden output from the dataset. Format: <output_len_mean>,<output_len_stdev>. E.g. 100,10 will randomize the output length with mean=100 and variance=10.

token-norm-dist#

Prepare synthetic dataset by generating random tokens with normal dist lengths.

Usage

trtllm-bench prepare-dataset token-norm-dist [OPTIONS]

Options

--num-requests <num_requests>#: Required Number of requests to be generated

--input-mean <input_mean>#: Required normal dist mean for input tokens

--input-stdev <input_stdev>#: Required normal dist stdev for input tokens

--output-mean <output_mean>#: Required normal dist mean for output tokens

--output-stdev <output_stdev>#: Required normal dist stdev for output tokens

token-unif-dist#

Prepare synthetic dataset by generating random tokens with normal uniformly lengths.

Usage

trtllm-bench prepare-dataset token-unif-dist [OPTIONS]

Options

--num-requests <num_requests>#: Required Number of requests to be generated

--input-min <input_min>#: Required uniform dist (inclusive) min for input tokens

--input-max <input_max>#: Required normal dist (inclusive) max for input tokens

--output-min <output_min>#: Required normal dist (inclusive) min for output tokens

--output-max <output_max>#: Required normal dist (inclusive) max for output tokens

Dataset preparation#

Dataset Types:

Real datasets from various sources
Synthetic datasets with normal or uniform token distributions
LoRA task-specific datasets

Key Features:

Tokenizer integration for proper text preprocessing — the tokenizer is resolved from the model passed to trtllm-bench --model
Configurable random seeds for reproducible results
Support for LoRA adapters and task IDs
Output in JSON format compatible with trtllm-bench