TorchLlmArgs

class tensorrt_llm.llmapi.TorchLlmArgs(

*,

model: str | ~pathlib.Path,

tokenizer: str | ~pathlib.Path | ~transformers.tokenization_utils_base.PreTrainedTokenizerBase | ~tensorrt_llm.tokenizer.tokenizer.TokenizerBase | None = None,

tokenizer_mode: ~typing.Literal['auto',

'slow'] = 'auto',

custom_tokenizer: str | None = None,

post_processor_hook: str | None = None,

skip_tokenizer_init: bool = False,

trust_remote_code: bool = False,

tensor_parallel_size: int = 1,

dtype: str = 'auto',

revision: str | None = None,

tokenizer_revision: str | None = None,

model_kwargs: ~typing.Dict[str,

~typing.Any] | None = None,

pipeline_parallel_size: int = 1,

context_parallel_size: int = 1,

gpus_per_node: int | None = None,

moe_cluster_parallel_size: int | None = None,

moe_tensor_parallel_size: int | None = None,

moe_expert_parallel_size: int | None = None,

enable_attention_dp: bool = False,

enable_lm_head_tp_in_adp: bool = False,

pp_partition: ~typing.List[int] | None = None,

cp_config: ~tensorrt_llm.llmapi.llm_args.CpConfig | None = None,

load_format: str | ~tensorrt_llm.llmapi.llm_args.LoadFormat = LoadFormat.AUTO,

enable_lora: bool = False,

lora_config: ~tensorrt_llm._torch.peft.lora.config.LoraConfig | None = None,

kv_cache_config: ~tensorrt_llm.llmapi.llm_args.KvCacheConfig = <factory>,

enable_chunked_prefill: bool = False,

guided_decoding_backend: ~typing.Literal['xgrammar',

'llguidance'] | None = None,

batched_logits_processor: object | None = None,

iter_stats_max_iterations: ~typing.Annotated[int | None,

~annotated_types.Ge(ge=-1)] = None,

request_stats_max_iterations: ~typing.Annotated[int | None,

~annotated_types.Ge(ge=-1)] = None,

peft_cache_config: ~tensorrt_llm.llmapi.llm_args.PeftCacheConfig | None = None,

scheduler_config: ~tensorrt_llm.llmapi.llm_args.SchedulerConfig = <factory>,

cache_transceiver_config: ~tensorrt_llm.llmapi.llm_args.CacheTransceiverConfig | None = None,

sparse_attention_config: ~typing.Annotated[~tensorrt_llm.llmapi.llm_args.QSASparseAttentionConfig | ~tensorrt_llm.llmapi.llm_args.RocketSparseAttentionConfig | ~tensorrt_llm.llmapi.llm_args.DeepSeekSparseAttentionConfig | ~tensorrt_llm.llmapi.llm_args.DeepSeekV4SparseAttentionConfig | ~tensorrt_llm.llmapi.llm_args.SkipSoftmaxAttentionConfig | ~tensorrt_llm.llmapi.llm_args.MiniMaxM3SparseAttentionConfig,

FieldInfo(annotation=NoneType,

required=True,

discriminator='algorithm')] | None = None,

kv_cache_compression_config: ~typing.Annotated[~tensorrt_llm.llmapi.llm_args.ColdPageQuantizationCompressionConfig | ~tensorrt_llm.llmapi.llm_args.TriAttentionKvCacheCompressionConfig,

FieldInfo(annotation=NoneType,

required=True,

discriminator='algorithm')] | None = None,

speculative_config: ~typing.Annotated[~tensorrt_llm.llmapi.llm_args.DraftTargetDecodingConfig | ~tensorrt_llm.llmapi.llm_args.Eagle3DecodingConfig | ~tensorrt_llm.llmapi.llm_args.EagleDecodingConfig | ~tensorrt_llm.llmapi.llm_args.MTPDecodingConfig | ~tensorrt_llm.llmapi.llm_args.NGramDecodingConfig | ~tensorrt_llm.llmapi.llm_args.SADecodingConfig | ~tensorrt_llm.llmapi.llm_args.UserProvidedDecodingConfig | ~tensorrt_llm.llmapi.llm_args.SaveHiddenStatesDecodingConfig | ~tensorrt_llm.llmapi.llm_args.PARDDecodingConfig | ~tensorrt_llm.llmapi.llm_args.DFlashDecodingConfig | ~tensorrt_llm.llmapi.llm_args.DSparkDecodingConfig | ~tensorrt_llm.llmapi.llm_args.AutoDecodingConfig,

FieldInfo(annotation=NoneType,

required=True,

discriminator='decoding_type')] | None = None,

max_batch_size: int | None = 2048,

max_input_len: int | None = 1024,

max_seq_len: int | None = None,

max_beam_width: int | None = 1,

max_num_tokens: int | None = 8192,

gather_generation_logits: bool = False,

num_postprocess_workers: int = 0,

postprocess_tokenizer_dir: str | None = None,

num_serve_frontends: ~typing.Annotated[int,

~annotated_types.Ge(ge=1),

~annotated_types.Le(le=64)] = 1,

reasoning_parser: str | None = None,

decoding_config: object | None = None,

_mpi_session: object | None = None,

otlp_traces_endpoint: str | None = None,

backend: ~typing.Literal['pytorch'] = 'pytorch',

return_perf_metrics: bool = False,

perf_metrics_output_dir: str | None = None,

perf_metrics_max_requests: ~typing.Annotated[int,

~annotated_types.Ge(ge=0)] = 0,

prometheus_metrics_config: ~tensorrt_llm.llmapi.llm_args.PrometheusMetricsConfig | None = None,

enable_energy_metrics: bool = False,

orchestrator_type: ~typing.Literal['rpc',

'ray'] | None = None,

env_overrides: ~typing.Dict[str,

str] | None = None,

telemetry_config: ~tensorrt_llm.usage.config.TelemetryConfig = <factory>,

generation_config: ~typing.Literal['auto',

'trtllm'] = 'trtllm',

garbage_collection_gen0_threshold: int = 20000,

cuda_graph_config: ~typing.Annotated[~tensorrt_llm.llmapi.llm_args.DecodeCudaGraphConfig | ~tensorrt_llm.llmapi.llm_args.EncodeCudaGraphConfig,

FieldInfo(annotation=NoneType,

required=True,

discriminator='mode')] | None = <factory>,

encoder_cuda_graph_config: ~tensorrt_llm.llmapi.llm_args.EncodeCudaGraphConfig | None = None,

enable_encoder_decoder_mixed_cuda_graph: bool = True,

multimodal_config: ~tensorrt_llm.llmapi.llm_args.MultimodalConfig = <factory>,

attention_dp_config: ~tensorrt_llm.llmapi.llm_args.AttentionDpConfig | None = None,

disable_overlap_scheduler: bool = False,

moe_config: ~tensorrt_llm.llmapi.llm_args.MoeConfig = <factory>,

nvfp4_gemm_config: ~tensorrt_llm.llmapi.llm_args.Nvfp4GemmConfig = <factory>,

dwdp_config: ~tensorrt_llm.llmapi.llm_args.DwdpConfig | None = None,

encoder_max_batch_size: int | None = None,

encoder_max_num_tokens: int | None = None,

attn_backend: str = 'TRTLLM',

enable_mla_skip_correction: bool = False,

mla_skip_correction_threshold: ~typing.Annotated[float,

~annotated_types.Gt(gt=0.0),

~annotated_types.Le(le=32.0)] = 8.0,

sampler_force_async_worker: bool = False,

enable_speculative_beam_history_d2h: bool = False,

enable_early_first_token_response: bool = False,

enable_low_latency_host_dispatch: bool = False,

enable_iter_perf_stats: bool = False,

enable_iter_req_stats: bool = False,

print_iter_log: bool = False,

batch_wait_timeout_ms: ~typing.Annotated[float,

~annotated_types.Ge(ge=0)] = 0,

batch_wait_timeout_iters: ~typing.Annotated[int,

~annotated_types.Ge(ge=0)] = 0,

batch_wait_max_tokens_ratio: ~typing.Annotated[float,

~annotated_types.Ge(ge=0),

~annotated_types.Le(le=1)] = 0,

torch_compile_config: ~tensorrt_llm.llmapi.llm_args.TorchCompileConfig | None = None,

prefill_cuda_graph_backend: ~tensorrt_llm.llmapi.llm_args.PrefillCudaGraphBackend = PrefillCudaGraphBackend.DISABLED,

prefill_capture_num_tokens: ~typing.List[int] | None = None,

enable_autotuner: bool = True,

use_fine_grained_sync: bool = False,

enable_layerwise_nvtx_marker: bool = False,

enable_min_latency: bool = False,

stream_interval: ~typing.Annotated[int,

~annotated_types.Gt(gt=0)] = 1,

force_dynamic_quantization: bool = False,

allreduce_strategy: ~typing.Literal['AUTO',

'NCCL',

'UB',

'MINLATENCY',

'ONESHOT',

'TWOSHOT',

'LOWPRECISION',

'MNNVL',

'NCCL_SYMMETRIC'] | None = 'AUTO',

checkpoint_loader: object | None = None,

checkpoint_format: str | None = None,

checkpoint_io_policy: ~typing.Literal['auto',

'native',

'rank_striped_read_ahead'] = 'auto',

mx_config: ~tensorrt_llm.llmapi.llm_args.ModelExpressConfig = <factory>,

gms_config: ~tensorrt_llm.llmapi.llm_args.GmsConfig = <factory>,

kv_connector_config: ~tensorrt_llm.llmapi.llm_args.KvCacheConnectorConfig | None = None,

mm_encoder_only: bool = False,

disable_mm_encoder: bool = False,

encode_only: bool = False,

ray_worker_extension_cls: str | None = None,

ray_placement_config: ~tensorrt_llm.llmapi.llm_args.RayPlacementConfig | None = None,

ray_worker_nsight_options: dict[str,

str] | None = None,

sleep_config: ~tensorrt_llm.llmapi.llm_args.SleepConfig | None = None,

reorder_policy_config: ~tensorrt_llm.llmapi.llm_args.ReorderRequestPolicyConfig | None = None,

enable_resource_governor: bool = False,

use_cute_dsl_blockscaling_mm: bool = False,

use_cute_dsl_blockscaling_bmm: bool = False,

use_cute_dsl_bf16_bmm: bool = False,

use_cute_dsl_bf16_gemm: bool = False,

max_stats_len: ~typing.Annotated[int,

~annotated_types.Ge(ge=-1)] = 1000,

layer_wise_benchmarks_config: ~tensorrt_llm.llmapi.llm_args.LayerwiseBenchmarksConfig = <factory>,

)

Bases: BaseLlmArgs

field allreduce_strategy: Literal['AUTO', 'NCCL', 'UB', 'MINLATENCY', 'ONESHOT', 'TWOSHOT', 'LOWPRECISION', 'MNNVL', 'NCCL_SYMMETRIC'] | None = 'AUTO'

beta Allreduce strategy to use.

field attention_dp_config: AttentionDpConfig | None = None

beta Optimized load-balancing for the DP Attention scheduler.

field attn_backend: str = 'TRTLLM'

beta Attention backend to use.

field backend: Literal['pytorch'] = 'pytorch'

deprecated The backend to use for this LLM instance.

field batch_wait_max_tokens_ratio: float = 0

prototype Token accumulation threshold ratio for batch scheduling optimization. If greater than 0, the scheduler will accumulate requests locally until the total token count reaches batch_wait_max_tokens_ratio * max_num_tokens. This mechanism enhances GPU utilization efficiency by ensuring adequate batch sizes. If 0, disables token-based batching delays.

Constraints:

• ge = 0

• le = 1

field batch_wait_timeout_iters: Annotated[int, Ge(ge=0)] = 0

prototype Maximum number of iterations the scheduler will wait to accumulate new coming requests for improved GPU utilization efficiency. If greater than 0, the scheduler will delay batch processing to gather more requests up to the specified iteration limit. If 0, disables timeout-iters-based batching delays.

Constraints:

• ge = 0

field batch_wait_timeout_ms: Annotated[float, Ge(ge=0)] = 0

prototype If greater than 0, the request queue might wait up to batch_wait_timeout_ms to receive max_batch_size requests, if fewer than max_batch_size requests are currently available. If 0, no waiting occurs.

Constraints:

• ge = 0

field batched_logits_processor: object | None = None

stable Batched logits processor.

field cache_transceiver_config: CacheTransceiverConfig | None = None

prototype Cache transceiver config.

field checkpoint_format: str | None = None

prototype The registered checkpoint loader format to use. MX selects ModelExpress as an opportunistic P2P loading path and falls back to loading the provided Hugging Face checkpoint; it does not require converting that checkpoint to an MX-specific format. You may use a custom checkpoint format by subclassing BaseCheckpointLoader and registering it with register_checkpoint_loader. If neither checkpoint_format nor checkpoint_loader are provided, checkpoint_format will be set to HF and the default HfCheckpointLoader will be used. If checkpoint_format and checkpoint_loader are both provided, checkpoint_loader will be ignored.

field checkpoint_io_policy: Literal['auto', 'native', 'rank_striped_read_ahead'] = 'auto'

prototype Controls checkpoint storage I/O independently of checkpoint format. ‘auto’ selects rank-striped read-ahead for compatible built-in PyTorch/HF loads and selects native I/O otherwise. ‘native’ preserves the existing loader. ‘rank_striped_read_ahead’ lets node-local ranks read disjoint SafeTensors extents while native mapping, materialization, and H2D continue. Incompatible configurations select native I/O before optimized reader or collective setup. Runtime-ineligible loads fall back to native before model mutation.

field checkpoint_loader: object | None = None

prototype The checkpoint loader to use for this LLM instance. You may use a custom checkpoint loader by subclassing BaseCheckpointLoader and providing an instance of the subclass here to load weights from a custom checkpoint format. If neither checkpoint_format nor checkpoint_loader are provided, checkpoint_format will be set to HF and the default HfCheckpointLoader will be used. If checkpoint_format and checkpoint_loader are both provided, checkpoint_loader will be ignored.

field context_parallel_size: int = 1

stable The context parallel size.

field cp_config: CpConfig | None = None

prototype Context parallel config.

field cuda_graph_config: Annotated[DecodeCudaGraphConfig | EncodeCudaGraphConfig, FieldInfo(annotation=NoneType, required=True, discriminator='mode')] | None [Optional]

beta CUDA graph config. If true, use CUDA graphs for decoding. CUDA graphs are only created for the batch sizes in cuda_graph_config.batch_sizes, and are enabled for batches that consist of decoding requests only (the reason is that it’s hard to capture a single graph with prefill requests since the input shapes are a function of the sequence lengths). Note that each CUDA graph can use up to 200 MB of extra memory.

field custom_tokenizer: str | None = None

prototype Specify a custom tokenizer implementation. Accepts either: (1) a built-in alias (e.g., ‘deepseek_v32’), or (2) a Python import path (e.g., ‘tensorrt_llm.tokenizer.deepseek_v32.DeepseekV32Tokenizer’). The tokenizer class must implement ‘from_pretrained(path, **kwargs)’ and the TokenizerBase interface.

field disable_mm_encoder: bool = False

prototype Skip instantiating and loading the multimodal (e.g. vision) encoder of a multimodal checkpoint and serve it text-only. Saves the encoder’s GPU memory (enlarging the KV cache pool) for workloads that never send image/video/audio inputs; such requests are rejected. Only takes effect for model implementations that support it (currently Mistral3 and the Qwen3-VL / Qwen3.5-VL family); a no-op otherwise. Defaults to False.

field disable_overlap_scheduler: bool = False

beta Disable the overlap scheduler.

field dtype: str = 'auto'

stable The data type to use for the model. When ‘auto’ (default), it is read from the HF config.json (‘dtype’, or the deprecated ‘torch_dtype’); for composite/VLM configs it falls back to the nested text_config.dtype. Defaults to bfloat16 if none is found.

field dwdp_config: DwdpConfig | None = None

prototype DWDP (Distributed Weight Data Parallelism) config.

field enable_attention_dp: bool = False

beta Enable attention data parallel.

field enable_autotuner: bool = True

prototype Enable autotuner for all tunable ops. This flag is for debugging purposes only, and the performance may significantly degrade if set to false.

field enable_chunked_prefill: bool = False

stable Enable chunked prefill.

field enable_early_first_token_response: bool = False

prototype Under the overlap scheduler, emit the first-token response ahead of the next sample step to reduce TTFT. No effect when the overlap scheduler is disabled.

field enable_encoder_decoder_mixed_cuda_graph: bool = True

prototype Enable the mixed-batch CUDA graph performance optimization for encoder-decoder models. The graph handles decoder iterations containing both context and generation requests. It is enabled by default when both cuda_graph_config and encoder_cuda_graph_config produce usable graph shapes.

field enable_energy_metrics: bool = False

prototype Enable GPU energy monitoring via NVML. When enabled, the server exposes an /energy_metrics endpoint that reports cumulative GPU energy consumption in joules.

field enable_iter_perf_stats: bool = False

prototype Enable iteration performance statistics.

field enable_iter_req_stats: bool = False

prototype If true, enables per request stats per iteration. Must also set enable_iter_perf_stats to true to get request stats.

field enable_layerwise_nvtx_marker: bool = False

beta If true, enable layerwise nvtx marker.

field enable_lm_head_tp_in_adp: bool = False

prototype Enable LM head TP in attention dp.

field enable_lora: bool = False

stable Enable LoRA.

field enable_low_latency_host_dispatch: bool = False

prototype Use low-latency spin-wait mode for CUDA host task dispatch (cudaLaunchHostFunc_v2 with cudaHostTaskSpinWait). Reduces callback latency at the cost of a CPU core spinning while waiting for the GPU event. Requires CUDA 13.2+; on older CUDA versions, falls back to the default blocking mode and logs a one-time warning.

field enable_min_latency: bool = False

beta If true, enable min-latency mode. Currently only used for Llama4.

field enable_mla_skip_correction: bool = False

prototype Enable threshold-based skip-correction for trtllm-gen MLA attention kernels on SM100 and SM103. When enabled, mla_skip_correction_threshold controls the optimization threshold.

field enable_resource_governor: bool = False

prototype Enable the resource governor for runtime cache management operations such as KV cache truncation. This adds a per-iteration broadcast collective.

field enable_speculative_beam_history_d2h: bool = False

prototype Opt-in beam-search optimization: skip per-step beam-history D2H copies on likely-non-terminal steps via a host-side predictor and route the remaining copies through a private side stream. Mispredictions fall back to a synchronous .cpu(), preserving correctness but breaking overlap on that step. Incompatible with the async D2H worker (sampler_force_async_worker=True or confidential compute).

field encode_only: bool = False

prototype Set to True to use the batch-forward encode() path, which runs a single forward pass and returns the model output directly, bypassing the scheduler and autoregressive loop. Works for encoder-only models (BERT, RoBERTa, reward models) and decoder models used in single-prefill mode (e.g., extracting embeddings). When False (default), uses the standard generate() path.

field encoder_cuda_graph_config: EncodeCudaGraphConfig | None = None

prototype CUDA graph configuration for the encoder forward pass of an encoder-decoder model. Use cuda_graph_config for the decoder and this field for the encoder. Encoder CUDA graphs require encoder_max_batch_size to be set.

field encoder_max_batch_size: int | None = None

prototype Maximum number of top-level encoder inputs processed in one iteration. For encoder-decoder models, each encoder request counts as one input. For multimodal models, each atomic image, video, or other encoder item counts as one input, even if it expands into multiple internal attention sequences. For encoder-decoder models, it also limits encoder CUDA graph batch sizes. Falls back to max_batch_size when unset.

field encoder_max_num_tokens: int | None = None

prototype Maximum number of encoder tokens. For encoder-decoder models, this limits encoder CUDA graph total-token buckets. For multimodal models, it limits encoder attention tokens scheduled in one iteration and is shared across all encoded modalities. It falls back to max_num_tokens when unset. Because an atomic multimodal item cannot be split, the effective budget is raised to the model’s largest atomic item when necessary.

field env_overrides: Dict[str, str] | None = None

prototype [EXPERIMENTAL] Environment variable overrides. NOTE: import-time-cached env vars in the code won’t update unless the code fetches them from os.environ on demand.

field force_dynamic_quantization: bool = False

prototype If true, force dynamic quantization. Defaults to False.

field garbage_collection_gen0_threshold: int = 20000

beta Threshold for Python garbage collection of generation 0 objects. Lower values trigger more frequent garbage collection.

field gather_generation_logits: bool = False

prototype Gather generation logits.

field generation_config: Literal['auto', 'trtllm'] = 'trtllm'

prototype Controls whether sampling defaults are loaded from the model’s generation_config.json. ‘auto’ applies supported values when the request does not specify them; ‘trtllm’ preserves TRT-LLM defaults. Precedence is request values, generation_config.json values, then TRT-LLM defaults.

field gms_config: GmsConfig [Optional]

prototype GPU Memory Service (GMS) weight sharing config.

field gpus_per_node: int | None = None

beta The number of GPUs per node.

field guided_decoding_backend: Literal['xgrammar', 'llguidance'] | None = None

stable Guided decoding backend. llguidance is supported in PyTorch backend only.

field iter_stats_max_iterations: int | None = None

prototype The maximum number of iterations for iter stats. Set to -1 to keep all iteration stats. Set to 0 to disable iteration stats in the TensorRT executor.

Constraints:

• ge = -1

field kv_cache_compression_config: KvCacheCompressionConfigType | None = None

prototype KV-cache compression config; None disables compression.

field kv_cache_config: KvCacheConfig [Optional]

stable KV cache config.

field kv_connector_config: KvCacheConnectorConfig | None = None

prototype The config for KV cache connector.

field layer_wise_benchmarks_config: LayerwiseBenchmarksConfig [Optional]

prototype Configuration for layer-wise benchmarks calibration.

field load_format: str | LoadFormat = LoadFormat.AUTO

stable How to load the model weights. By default, detect the weight type from the model checkpoint.

field lora_config: LoraConfig | None = None

stable LoRA configuration for the model.

field max_batch_size: int | None = 2048

stable The maximum batch size.

field max_beam_width: int | None = 1

stable The maximum beam width.

field max_input_len: int | None = 1024

stable The maximum input length.

field max_num_tokens: int | None = 8192

stable The maximum number of tokens.

field max_seq_len: int | None = None

stable The maximum sequence length.

field max_stats_len: int = 1000

prototype The max number of performance statistic entries. Set to -1 to keep all entries. Set to 0 to use a minimum buffer size of 1.

Constraints:

• ge = -1

field mla_skip_correction_threshold: float = 8.0

prototype Threshold for threshold-based skip-correction. This is used only when enable_mla_skip_correction is True. The default is 8. The maximum supported value depends on the selected kernel’s BMM2 dtype: 8 for E4M3, 15 for FP16, and 32 for BF16.

Constraints:

• gt = 0.0

• le = 32.0

field mm_encoder_only: bool = False

prototype Only load/execute the vision encoder part of the full model. Defaults to False.

field model: str | Path [Required]

stable The path to the model checkpoint or the model name from the Hugging Face Hub.

field model_kwargs: Dict[str, Any] | None = None

prototype Optional parameters overriding model config defaults. Precedence: (1) model_kwargs, (2) model config file, (3) model config class defaults. Unknown keys are ignored

field moe_cluster_parallel_size: int | None = None

deprecated The cluster parallel size for MoE model’s expert weights.

field moe_config: MoeConfig [Optional]

beta MoE config.

field moe_expert_parallel_size: int | None = None

stable The expert parallel size for MoE model’s expert weights.

field moe_tensor_parallel_size: int | None = None

stable The tensor parallel size for MoE model’s expert weights.

field mpi_session: object | None = None (alias '_mpi_session')

stable The optional MPI session to use for this LLM instance.

field multimodal_config: MultimodalConfig [Optional]

prototype Multimodal model configuration.

field mx_config: ModelExpressConfig [Optional]

prototype ModelExpress (MX) P2P checkpoint loading config.

field num_postprocess_workers: int = 0

prototype The number of processes used for postprocessing the generated tokens, including detokenization.

field num_serve_frontends: int = 1

prototype The number of HTTP frontend processes serving one executor. Used by trtllm-serve: values > 1 run additional attached frontend processes that share the serving port via SO_REUSEPORT (classic IPC executor path only).

Constraints:

• ge = 1

• le = 64

field nvfp4_gemm_config: Nvfp4GemmConfig [Optional]

beta NVFP4 GEMM backend config.

field orchestrator_type: Literal['rpc', 'ray'] | None = None

prototype The orchestrator type to use. Defaults to None, which uses MPI.

field otlp_traces_endpoint: str | None = None

prototype Target URL to which OpenTelemetry traces will be sent.

field peft_cache_config: PeftCacheConfig | None = None

prototype PEFT cache config.

field perf_metrics_max_requests: NonNegativeInt = 0

deprecated Deprecated compatibility field. Completed per-request metrics are no longer retained in memory.

Constraints:

• ge = 0

field perf_metrics_output_dir: str | None = None

prototype Directory for per-process performance metrics JSONL files. Setting this enables collection even when return_perf_metrics is false.

field pipeline_parallel_size: int = 1

stable The pipeline parallel size.

field post_processor_hook: str | None = None

prototype Python import path of a user post-processing hook applied after detokenization and before the per-endpoint response formatter (e.g. ‘my_pkg.guardrail.MyPostProcessorHook’). The class must be importable and picklable, take no constructor arguments, and be callable as ‘__call__(chunk) -> verdict’ (see tensorrt_llm.executor.postprocessor_hook). It runs once per output, per streaming chunk, and may rewrite, suppress, or terminate the output; it owns its own per-request state.

field postprocess_tokenizer_dir: str | None = None

prototype The path to the tokenizer directory for postprocessing.

field pp_partition: List[int] | None = None

prototype Pipeline parallel partition, a list of each rank’s layer number.

field prefill_capture_num_tokens: List[int] | None = None

prototype Token-count buckets captured by the selected prefill CUDA graph implementation.

field prefill_cuda_graph_backend: PrefillCudaGraphBackend = PrefillCudaGraphBackend.DISABLED

prototype CUDA graph implementation used for prefill requests. Defaults to disabled.

field print_iter_log: bool = False

beta Print iteration logs.

field prometheus_metrics_config: PrometheusMetricsConfig | None = None

prototype Configuration for Prometheus metrics collection, including custom histogram bucket boundaries.

field ray_placement_config: RayPlacementConfig | None = None

prototype Placement config for RayGPUWorker. Only used with AsyncLLM and orchestrator_type=’ray’.

field ray_worker_extension_cls: str | None = None

prototype The full worker extension class name including module path. Allows users to extend the functions of the RayGPUWorker class.

field ray_worker_nsight_options: dict[str, str] | None = None

prototype Nsight options.

field reasoning_parser: str | None = None

prototype The parser to separate reasoning content from output.

field reorder_policy_config: ReorderRequestPolicyConfig | None = None

prototype The request reordering policy to use.

field request_stats_max_iterations: int | None = None

prototype The maximum number of iterations for request stats. Set to -1 to keep all request stats. Set to 0 to disable request stats.

Constraints:

• ge = -1

field return_perf_metrics: bool = False

prototype Allow serving responses to include per-request performance metrics when the request sets X-TRTLLM-return-metrics: 1.

field revision: str | None = None

stable The revision to use for the model.

field sampler_force_async_worker: bool = False

prototype Force usage of the async worker in the sampler for D2H copies, even if confidential compute is not active. Normally, the async worker should only be used when confidential compute is active. This argument is provided to enable it for testing purposes, irrespective of confidential compute state.

field scheduler_config: SchedulerConfig [Optional]

prototype Scheduler config.

field skip_tokenizer_init: bool = False

stable Whether to skip the tokenizer initialization.

field sleep_config: SleepConfig | None = None

prototype Configuration for the LLM sleep feature. Sleep feature requires extra setup that may slow down model loading. Only enable it if you intend to use this feature.

field sparse_attention_config: SparseAttentionConfig | None = None

prototype Sparse attention config.

field speculative_config: SpeculativeConfig | None = None

stable Speculative decoding config.

field stream_interval: Annotated[int, Gt(gt=0)] = 1

stable The iteration interval to create responses under the streaming mode. Set this to a larger value when the batch size is large, which helps reduce the streaming overhead.

Constraints:

• gt = 0

field telemetry_config: TelemetryConfig [Optional]

prototype Telemetry configuration (opt-out, usage context).

field tensor_parallel_size: int = 1

stable The tensor parallel size.

field tokenizer: str | Path | TokenizerBase | PreTrainedTokenizerBase | None = None

stable The path to the tokenizer checkpoint or the tokenizer name from the Hugging Face Hub.

field tokenizer_mode: Literal['auto', 'slow'] = 'auto'

stable The mode to initialize the tokenizer.

field tokenizer_revision: str | None = None

stable The revision to use for the tokenizer.

field torch_compile_config: TorchCompileConfig | None = None

prototype Torch compile config.

field trust_remote_code: bool = False

stable Whether to trust the remote code.

field use_cute_dsl_bf16_bmm: bool = False

prototype If true, use CuTe DSL bf16 persistent GEMM for BMM on Blackwell.

field use_cute_dsl_bf16_gemm: bool = False

prototype If true, use CuTe DSL bf16 persistent GEMM for Linear layers on Blackwell.

field use_cute_dsl_blockscaling_bmm: bool = False

prototype If true, use CuTe DSL fp8 blockscaling bmm implementation.

field use_cute_dsl_blockscaling_mm: bool = False

prototype If true, use CuTe DSL fp8 blockscaling mm implementation.

field use_fine_grained_sync: bool = False

prototype Enable fine-grained synchronization for MoE kernels on SM107. The FC1 producer kernel signals per-tile completion flags in device memory and the FC2 consumer kernel waits on them, so the two GEMMs overlap instead of serializing at kernel launch boundaries.

class Config

Bases: object

extra = 'forbid'

__init__(**data: Any) → None

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

validator coerce_env_overrides_to_str  »  env_overrides

Coerce env_overrides values to strings for os.environ compatibility.

validator convert_load_format  »  load_format

classmethod from_yaml(yaml_path: str | Path)

get_runtime_sizes() → Tuple[int, int, int, int]

validator infer_cuda_graph_config_mode  »  cuda_graph_config

validator init_multimodal_config  »  multimodal_config

validator normalize_disabled_mm_encoder_cache  »  all fields

validator normalize_max_stats_len  »  max_stats_len

validator normalize_optional_fields_to_defaults  »  all fields

Normalize certain fields to their declared default values in case a user explicitly sets them to None.

This is necessary because downstream code expects these fields to be non-None. At the same time, we still need to accept None as a valid value to avoid a breaking change.

validator normalize_prefill_cuda_graph_config  »  all fields

Normalize legacy piecewise CUDA graph options into prefill fields.

validator sync_quant_config_with_kv_cache_config_dtype  »  all fields

validator validate_and_init_tokenizer  »  all fields

Initialize tokenizer based on configuration.

validator validate_checkpoint_format  »  all fields

validator validate_cute_dsl_bf16  »  all fields

validator validate_early_first_token_response  »  all fields

validator validate_encode_only_torch_compile_config  »  all fields

validator validate_encoder_cuda_graph_config  »  all fields

validator validate_encoder_modes  »  all fields

validator validate_encoder_runtime_sizes  »  encoder_max_batch_size, encoder_max_num_tokens

validator validate_gms_config  »  all fields

Warn when GMS settings are provided without enabling GMS load.

Catches the most common misconfiguration: a user customizes gms_config (e.g. sets a custom socket_path or mode) but forgets to set load_format='GMS', in which case the entire GMS config is silently ignored. We emit a warning so the user notices at config time instead of debugging “why are my workers not zero-copy sharing weights?” afterwards.

Detection is by deviation from defaults: any of socket_path != None, mode != 'auto', or tag != 'weights' triggers the warning when load_format != LoadFormat.GMS. This is intentionally a warning (not an error) so callers that pre-populate config objects from templates aren’t broken.

Returns:

self (Pydantic model_validator contract).

validator validate_gms_moe_compat  »  all fields

Reject LoadFormat.GMS combined with a MoE load balancer.

The MoeLoadBalancer’s register_weight_slots_after_to_cuda and finalize_model run AFTER the GMS RW pool is closed and finalize_write has committed, so any CUDA allocations they make land in non-GMS memory and are NOT part of the committed layout that RO peers receive. The result is “wrong inference, no error” on RO peers (broken MoE routing state). Failing at config-validation time is strictly better than that silent miscompute.

The fix for this gap (running the MoE finalize work INSIDE mem_pool_scope and BEFORE finalize_write so MoE allocations are part of the committed layout) is tracked as the (MoE, GMS) follow-up; see model_loader.py’s TODO(GMS-MOE-LB) comment.

Returns:

self (Pydantic model_validator contract).

Raises:

ValueError – When load_format == LoadFormat.GMS and moe_config.load_balancer is set.

validator validate_gpus_per_node  »  gpus_per_node

validator validate_helix_tokens_per_block  »  all fields

Validate that cp_config.tokens_per_block matches kv_cache_config.tokens_per_block when HELIX parallelism is active.

validator validate_load_balancer  »  all fields

validator validate_lora_config_consistency  »  all fields

validator validate_mla_skip_correction_config  »  all fields

validator validate_mx_config  »  all fields

validator validate_parallel_config  »  all fields

validator validate_peft_cache_config  »  all fields

validator validate_ray_placement_config  »  all fields

validator validate_ray_worker_extension_cls  »  all fields

validator validate_runtime_args  »  all fields

validator validate_speculative_beam_history_d2h  »  all fields

validator validate_speculative_config  »  all fields

validator warn_non_pytorch_checkpoint_io_policy_fallback  »  all fields

warn_on_unstable_feature_usage() → TorchLlmArgs

Warn on unstable feature usage.

decoding_config: object | None

Read-only data descriptor used to emit a runtime deprecation warning before accessing a deprecated field.

msg

The deprecation message to be emitted.

wrapped_property

The property instance if the deprecated field is a computed field, or None.

field_name

The name of the field being deprecated.

property extra_resource_managers: Dict[str, object]

property is_partial_model_loading: bool

Whether model overrides request loading only part of the model.

property parallel_config: _ParallelConfig

property quant_config: QuantConfig

property speculative_model: str | Path | None