LLM#

class tensorrt_llm.llmapi.LLM(

model: str | Path,

tokenizer: str | Path | TokenizerBase | PreTrainedTokenizerBase | None = None,

tokenizer_mode: Literal['auto', 'slow'] = 'auto',

skip_tokenizer_init: bool = False,

trust_remote_code: bool = False,

tensor_parallel_size: int = 1,

dtype: str = 'auto',

revision: str | None = None,

tokenizer_revision: str | None = None,

**kwargs: Any,

)[source]#

Bases: _TorchLLM

LLM class is the main class for running a LLM model.

For more details about the arguments, please refer to TorchLlmArgs.

Parameters:

model (Union[str, pathlib.Path]) – stable The path to the model checkpoint or the model name from the Hugging Face Hub.
tokenizer (Union[str, pathlib.Path, transformers.tokenization_utils_base.PreTrainedTokenizerBase, tensorrt_llm.tokenizer.tokenizer.TokenizerBase, NoneType]) – stable The path to the tokenizer checkpoint or the tokenizer name from the Hugging Face Hub. Defaults to None.
tokenizer_mode (Literal['auto', 'slow']) – stable The mode to initialize the tokenizer. Defaults to auto.
custom_tokenizer (Optional[str]) – prototype Specify a custom tokenizer implementation. Accepts either: (1) a built-in alias (e.g., ‘deepseek_v32’), or (2) a Python import path (e.g., ‘tensorrt_llm.tokenizer.deepseek_v32.DeepseekV32Tokenizer’). The tokenizer class must implement ‘from_pretrained(path, **kwargs)’ and the TokenizerBase interface. Defaults to None.
post_processor_hook (Optional[str]) – prototype Python import path of a user post-processing hook applied after detokenization and before the per-endpoint response formatter (e.g. ‘my_pkg.guardrail.MyPostProcessorHook’). The class must be importable and picklable, take no constructor arguments, and be callable as ‘__call__(chunk) -> verdict’ (see tensorrt_llm.executor.postprocessor_hook). It runs once per output, per streaming chunk, and may rewrite, suppress, or terminate the output; it owns its own per-request state. Defaults to None.
skip_tokenizer_init (bool) – stable Whether to skip the tokenizer initialization. Defaults to False.
trust_remote_code (bool) – stable Whether to trust the remote code. Defaults to False.
tensor_parallel_size (int) – stable The tensor parallel size. Defaults to 1.
dtype (str) – stable The data type to use for the model. When ‘auto’ (default), it is read from the HF config.json (‘dtype’, or the deprecated ‘torch_dtype’); for composite/VLM configs it falls back to the nested text_config.dtype. Defaults to bfloat16 if none is found. Defaults to auto.
revision (Optional[str]) – stable The revision to use for the model. Defaults to None.
tokenizer_revision (Optional[str]) – stable The revision to use for the tokenizer. Defaults to None.
model_kwargs (Optional[Dict[str, Any]]) – prototype Optional parameters overriding model config defaults. Precedence: (1) model_kwargs, (2) model config file, (3) model config class defaults. Unknown keys are ignored Defaults to None.
pipeline_parallel_size (int) – stable The pipeline parallel size. Defaults to 1.
context_parallel_size (int) – stable The context parallel size. Defaults to 1.
gpus_per_node (Optional[int]) – beta The number of GPUs per node. Defaults to None.
moe_tensor_parallel_size (Optional[int]) – stable The tensor parallel size for MoE model’s expert weights. Defaults to None.
moe_expert_parallel_size (Optional[int]) – stable The expert parallel size for MoE model’s expert weights. Defaults to None.
enable_attention_dp (bool) – beta Enable attention data parallel. Defaults to False.
enable_lm_head_tp_in_adp (bool) – prototype Enable LM head TP in attention dp. Defaults to False.
pp_partition (Optional[List[int]]) – prototype Pipeline parallel partition, a list of each rank’s layer number. Defaults to None.
cp_config (Optional[tensorrt_llm.llmapi.llm_args.CpConfig]) – prototype Context parallel config. Defaults to None.
load_format (Union[str, tensorrt_llm.llmapi.llm_args.LoadFormat]) – stable How to load the model weights. By default, detect the weight type from the model checkpoint. Defaults to 0.
enable_lora (bool) – stable Enable LoRA. Defaults to False.
lora_config (Optional[tensorrt_llm._torch.peft.lora.config.LoraConfig]) – stable LoRA configuration for the model. Defaults to None.
kv_cache_config (tensorrt_llm.llmapi.llm_args.KvCacheConfig) – stable KV cache config. Defaults to None.
enable_chunked_prefill (bool) – stable Enable chunked prefill. Defaults to False.
guided_decoding_backend (Optional[Literal['xgrammar', 'llguidance']]) – stable Guided decoding backend. llguidance is supported in PyTorch backend only. Defaults to None.
batched_logits_processor (Optional[tensorrt_llm.sampling_params.BatchedLogitsProcessor]) – stable Batched logits processor. Defaults to None.
iter_stats_max_iterations (Optional[int]) – prototype The maximum number of iterations for iter stats. Set to -1 to keep all iteration stats. Set to 0 to disable iteration stats in the TensorRT executor. Defaults to None.
request_stats_max_iterations (Optional[int]) – prototype The maximum number of iterations for request stats. Set to -1 to keep all request stats. Set to 0 to disable request stats. Defaults to None.
peft_cache_config (Optional[tensorrt_llm.llmapi.llm_args.PeftCacheConfig]) – prototype PEFT cache config. Defaults to None.
scheduler_config (tensorrt_llm.llmapi.llm_args.SchedulerConfig) – prototype Scheduler config. Defaults to None.
cache_transceiver_config (Optional[tensorrt_llm.llmapi.llm_args.CacheTransceiverConfig]) – prototype Cache transceiver config. Defaults to None.
sparse_attention_config (Union[tensorrt_llm.llmapi.llm_args.QSASparseAttentionConfig, tensorrt_llm.llmapi.llm_args.RocketSparseAttentionConfig, tensorrt_llm.llmapi.llm_args.DeepSeekSparseAttentionConfig, tensorrt_llm.llmapi.llm_args.DeepSeekV4SparseAttentionConfig, tensorrt_llm.llmapi.llm_args.SkipSoftmaxAttentionConfig, tensorrt_llm.llmapi.llm_args.MiniMaxM3SparseAttentionConfig, NoneType]) – prototype Sparse attention config. Defaults to None.
kv_cache_compression_config (Union[tensorrt_llm.llmapi.llm_args.ColdPageQuantizationCompressionConfig, tensorrt_llm.llmapi.llm_args.TriAttentionKvCacheCompressionConfig, NoneType]) – prototype KV-cache compression config; None disables compression. Defaults to None.
speculative_config (Union[tensorrt_llm.llmapi.llm_args.DraftTargetDecodingConfig, tensorrt_llm.llmapi.llm_args.Eagle3DecodingConfig, tensorrt_llm.llmapi.llm_args.EagleDecodingConfig, tensorrt_llm.llmapi.llm_args.MTPDecodingConfig, tensorrt_llm.llmapi.llm_args.NGramDecodingConfig, tensorrt_llm.llmapi.llm_args.SADecodingConfig, tensorrt_llm.llmapi.llm_args.UserProvidedDecodingConfig, tensorrt_llm.llmapi.llm_args.SaveHiddenStatesDecodingConfig, tensorrt_llm.llmapi.llm_args.PARDDecodingConfig, tensorrt_llm.llmapi.llm_args.DFlashDecodingConfig, tensorrt_llm.llmapi.llm_args.DSparkDecodingConfig, tensorrt_llm.llmapi.llm_args.AutoDecodingConfig, NoneType]) – stable Speculative decoding config. Defaults to None.
max_batch_size (Optional[int]) – stable The maximum batch size. Defaults to 2048.
max_input_len (Optional[int]) – stable The maximum input length. Defaults to 1024.
max_seq_len (Optional[int]) – stable The maximum sequence length. Defaults to None.
max_beam_width (Optional[int]) – stable The maximum beam width. Defaults to 1.
max_num_tokens (Optional[int]) – stable The maximum number of tokens. Defaults to 8192.
gather_generation_logits (bool) – prototype Gather generation logits. Defaults to False.
num_postprocess_workers (int) – prototype The number of processes used for postprocessing the generated tokens, including detokenization. Defaults to 0.
postprocess_tokenizer_dir (Optional[str]) – prototype The path to the tokenizer directory for postprocessing. Defaults to None.
num_serve_frontends (int) – prototype The number of HTTP frontend processes serving one executor. Used by trtllm-serve: values > 1 run additional attached frontend processes that share the serving port via SO_REUSEPORT (classic IPC executor path only). Defaults to 1.
reasoning_parser (Optional[str]) – prototype The parser to separate reasoning content from output. Defaults to None.
otlp_traces_endpoint (Optional[str]) – prototype Target URL to which OpenTelemetry traces will be sent. Defaults to None.
return_perf_metrics (bool) – prototype Allow serving responses to include per-request performance metrics when the request sets X-TRTLLM-return-metrics: 1. Defaults to False.
perf_metrics_output_dir (Optional[str]) – prototype Directory for per-process performance metrics JSONL files. Setting this enables collection even when return_perf_metrics is false. Defaults to None.
prometheus_metrics_config (Optional[tensorrt_llm.llmapi.llm_args.PrometheusMetricsConfig]) – prototype Configuration for Prometheus metrics collection, including custom histogram bucket boundaries. Defaults to None.
enable_energy_metrics (bool) – prototype Enable GPU energy monitoring via NVML. When enabled, the server exposes an /energy_metrics endpoint that reports cumulative GPU energy consumption in joules. Defaults to False.
orchestrator_type (Optional[Literal['rpc', 'ray']]) – prototype The orchestrator type to use. Defaults to None, which uses MPI. Defaults to None.
env_overrides (Optional[Dict[str, str]]) – prototype [EXPERIMENTAL] Environment variable overrides. NOTE: import-time-cached env vars in the code won’t update unless the code fetches them from os.environ on demand. Defaults to None.
telemetry_config (tensorrt_llm.usage.config.TelemetryConfig) – prototype Telemetry configuration (opt-out, usage context). Defaults to None.
generation_config (Literal['auto', 'trtllm']) – prototype Controls whether sampling defaults are loaded from the model’s generation_config.json. ‘auto’ applies supported values when the request does not specify them; ‘trtllm’ preserves TRT-LLM defaults. Precedence is request values, generation_config.json values, then TRT-LLM defaults. Defaults to trtllm.
garbage_collection_gen0_threshold (int) – beta Threshold for Python garbage collection of generation 0 objects. Lower values trigger more frequent garbage collection. Defaults to 20000.
cuda_graph_config (Union[tensorrt_llm.llmapi.llm_args.DecodeCudaGraphConfig, tensorrt_llm.llmapi.llm_args.EncodeCudaGraphConfig, NoneType]) – beta CUDA graph config. If true, use CUDA graphs for decoding. CUDA graphs are only created for the batch sizes in cuda_graph_config.batch_sizes, and are enabled for batches that consist of decoding requests only (the reason is that it’s hard to capture a single graph with prefill requests since the input shapes are a function of the sequence lengths). Note that each CUDA graph can use up to 200 MB of extra memory. Defaults to None.
encoder_cuda_graph_config (Optional[tensorrt_llm.llmapi.llm_args.EncodeCudaGraphConfig]) – prototype CUDA graph configuration for the encoder forward pass of an encoder-decoder model. Use cuda_graph_config for the decoder and this field for the encoder. Encoder CUDA graphs require encoder_max_batch_size to be set. Defaults to None.
enable_encoder_decoder_mixed_cuda_graph (bool) – prototype Enable the mixed-batch CUDA graph performance optimization for encoder-decoder models. The graph handles decoder iterations containing both context and generation requests. It is enabled by default when both cuda_graph_config and encoder_cuda_graph_config produce usable graph shapes. Defaults to True.
multimodal_config (tensorrt_llm.llmapi.llm_args.MultimodalConfig) – prototype Defaults to None.
attention_dp_config (Optional[tensorrt_llm.llmapi.llm_args.AttentionDpConfig]) – beta Optimized load-balancing for the DP Attention scheduler. Defaults to None.
disable_overlap_scheduler (bool) – beta Disable the overlap scheduler. Defaults to False.
moe_config (tensorrt_llm.llmapi.llm_args.MoeConfig) – beta MoE config. Defaults to None.
nvfp4_gemm_config (tensorrt_llm.llmapi.llm_args.Nvfp4GemmConfig) – beta NVFP4 GEMM backend config. Defaults to None.
dwdp_config (Optional[tensorrt_llm.llmapi.llm_args.DwdpConfig]) – prototype DWDP (Distributed Weight Data Parallelism) config. Defaults to None.
encoder_max_batch_size (Optional[int]) – prototype Maximum number of top-level encoder inputs processed in one iteration. For encoder-decoder models, each encoder request counts as one input. For multimodal models, each atomic image, video, or other encoder item counts as one input, even if it expands into multiple internal attention sequences. For encoder-decoder models, it also limits encoder CUDA graph batch sizes. Falls back to max_batch_size when unset. Defaults to None.
encoder_max_num_tokens (Optional[int]) – prototype Maximum number of encoder tokens. For encoder-decoder models, this limits encoder CUDA graph total-token buckets. For multimodal models, it limits encoder attention tokens scheduled in one iteration and is shared across all encoded modalities. It falls back to max_num_tokens when unset. Because an atomic multimodal item cannot be split, the effective budget is raised to the model’s largest atomic item when necessary. Defaults to None.
attn_backend (str) – beta Attention backend to use. Defaults to TRTLLM.
enable_mla_skip_correction (bool) – prototype Enable threshold-based skip-correction for trtllm-gen MLA attention kernels on SM100 and SM103. When enabled, mla_skip_correction_threshold controls the optimization threshold. Defaults to False.
mla_skip_correction_threshold (float) – prototype Threshold for threshold-based skip-correction. This is used only when enable_mla_skip_correction is True. The default is 8. The maximum supported value depends on the selected kernel’s BMM2 dtype: 8 for E4M3, 15 for FP16, and 32 for BF16. Defaults to 8.0.
sampler_force_async_worker (bool) – prototype Force usage of the async worker in the sampler for D2H copies, even if confidential compute is not active. Normally, the async worker should only be used when confidential compute is active. This argument is provided to enable it for testing purposes, irrespective of confidential compute state. Defaults to False.
enable_speculative_beam_history_d2h (bool) – prototype Opt-in beam-search optimization: skip per-step beam-history D2H copies on likely-non-terminal steps via a host-side predictor and route the remaining copies through a private side stream. Mispredictions fall back to a synchronous .cpu(), preserving correctness but breaking overlap on that step. Incompatible with the async D2H worker (sampler_force_async_worker=True or confidential compute). Defaults to False.
enable_early_first_token_response (bool) – prototype Under the overlap scheduler, emit the first-token response ahead of the next sample step to reduce TTFT. No effect when the overlap scheduler is disabled. Defaults to False.
enable_low_latency_host_dispatch (bool) – prototype Use low-latency spin-wait mode for CUDA host task dispatch (cudaLaunchHostFunc_v2 with cudaHostTaskSpinWait). Reduces callback latency at the cost of a CPU core spinning while waiting for the GPU event. Requires CUDA 13.2+; on older CUDA versions, falls back to the default blocking mode and logs a one-time warning. Defaults to False.
enable_iter_perf_stats (bool) – prototype Enable iteration performance statistics. Defaults to False.
enable_iter_req_stats (bool) – prototype If true, enables per request stats per iteration. Must also set enable_iter_perf_stats to true to get request stats. Defaults to False.
print_iter_log (bool) – beta Print iteration logs. Defaults to False.
batch_wait_timeout_ms (float) – prototype If greater than 0, the request queue might wait up to batch_wait_timeout_ms to receive max_batch_size requests, if fewer than max_batch_size requests are currently available. If 0, no waiting occurs. Defaults to 0.
batch_wait_timeout_iters (int) – prototype Maximum number of iterations the scheduler will wait to accumulate new coming requests for improved GPU utilization efficiency. If greater than 0, the scheduler will delay batch processing to gather more requests up to the specified iteration limit. If 0, disables timeout-iters-based batching delays. Defaults to 0.
batch_wait_max_tokens_ratio (float) – prototype Token accumulation threshold ratio for batch scheduling optimization. If greater than 0, the scheduler will accumulate requests locally until the total token count reaches batch_wait_max_tokens_ratio * max_num_tokens. This mechanism enhances GPU utilization efficiency by ensuring adequate batch sizes. If 0, disables token-based batching delays. Defaults to 0.
torch_compile_config (Optional[tensorrt_llm.llmapi.llm_args.TorchCompileConfig]) – prototype Torch compile config. Defaults to None.
prefill_cuda_graph_backend (PrefillCudaGraphBackend) – prototype CUDA graph implementation used for prefill requests. Defaults to disabled. Defaults to disabled.
prefill_capture_num_tokens (Optional[List[int]]) – prototype Token-count buckets captured by the selected prefill CUDA graph implementation. Defaults to None.
enable_autotuner (bool) – prototype Enable autotuner for all tunable ops. This flag is for debugging purposes only, and the performance may significantly degrade if set to false. Defaults to True.
use_fine_grained_sync (bool) – prototype Enable fine-grained synchronization for MoE kernels on SM107. The FC1 producer kernel signals per-tile completion flags in device memory and the FC2 consumer kernel waits on them, so the two GEMMs overlap instead of serializing at kernel launch boundaries. Defaults to False.
enable_layerwise_nvtx_marker (bool) – beta If true, enable layerwise nvtx marker. Defaults to False.
enable_min_latency (bool) – beta If true, enable min-latency mode. Currently only used for Llama4. Defaults to False.
stream_interval (int) – stable The iteration interval to create responses under the streaming mode. Set this to a larger value when the batch size is large, which helps reduce the streaming overhead. Defaults to 1.
force_dynamic_quantization (bool) – prototype If true, force dynamic quantization. Defaults to False. Defaults to False.
allreduce_strategy (Optional[Literal['AUTO', 'NCCL', 'UB', 'MINLATENCY', 'ONESHOT', 'TWOSHOT', 'LOWPRECISION', 'MNNVL', 'NCCL_SYMMETRIC']]) – beta Allreduce strategy to use. Defaults to AUTO.
checkpoint_loader (Optional[tensorrt_llm._torch.models.checkpoints.BaseCheckpointLoader]) – prototype The checkpoint loader to use for this LLM instance. You may use a custom checkpoint loader by subclassing BaseCheckpointLoader and providing an instance of the subclass here to load weights from a custom checkpoint format. If neither checkpoint_format nor checkpoint_loader are provided, checkpoint_format will be set to HF and the default HfCheckpointLoader will be used. If checkpoint_format and checkpoint_loader are both provided, checkpoint_loader will be ignored. Defaults to None.
checkpoint_format (Optional[str]) – prototype The registered checkpoint loader format to use. MX selects ModelExpress as an opportunistic P2P loading path and falls back to loading the provided Hugging Face checkpoint; it does not require converting that checkpoint to an MX-specific format. You may use a custom checkpoint format by subclassing BaseCheckpointLoader and registering it with register_checkpoint_loader. If neither checkpoint_format nor checkpoint_loader are provided, checkpoint_format will be set to HF and the default HfCheckpointLoader will be used. If checkpoint_format and checkpoint_loader are both provided, checkpoint_loader will be ignored. Defaults to None.
checkpoint_io_policy (Literal['auto', 'native', 'rank_striped_read_ahead']) – prototype Controls checkpoint storage I/O independently of checkpoint format. ‘auto’ selects rank-striped read-ahead for compatible built-in PyTorch/HF loads and selects native I/O otherwise. ‘native’ preserves the existing loader. ‘rank_striped_read_ahead’ lets node-local ranks read disjoint SafeTensors extents while native mapping, materialization, and H2D continue. Incompatible configurations select native I/O before optimized reader or collective setup. Runtime-ineligible loads fall back to native before model mutation. Defaults to auto.
mx_config (tensorrt_llm.llmapi.llm_args.ModelExpressConfig) – prototype ModelExpress (MX) P2P checkpoint loading config. Defaults to None.
gms_config (tensorrt_llm.llmapi.llm_args.GmsConfig) – prototype GPU Memory Service (GMS) weight sharing config. Defaults to None.
kv_connector_config (Optional[tensorrt_llm.llmapi.llm_args.KvCacheConnectorConfig]) – prototype The config for KV cache connector. Defaults to None.
mm_encoder_only (bool) – prototype Only load/execute the vision encoder part of the full model. Defaults to False. Defaults to False.
disable_mm_encoder (bool) – prototype Skip instantiating and loading the multimodal (e.g. vision) encoder of a multimodal checkpoint and serve it text-only. Saves the encoder’s GPU memory (enlarging the KV cache pool) for workloads that never send image/video/audio inputs; such requests are rejected. Only takes effect for model implementations that support it (currently Mistral3 and the Qwen3-VL / Qwen3.5-VL family); a no-op otherwise. Defaults to False. Defaults to False.
encode_only (bool) – prototype Set to True to use the batch-forward encode() path, which runs a single forward pass and returns the model output directly, bypassing the scheduler and autoregressive loop. Works for encoder-only models (BERT, RoBERTa, reward models) and decoder models used in single-prefill mode (e.g., extracting embeddings). When False (default), uses the standard generate() path. Defaults to False.
ray_worker_extension_cls (Optional[str]) – prototype The full worker extension class name including module path. Allows users to extend the functions of the RayGPUWorker class. Defaults to None.
ray_placement_config (Optional[tensorrt_llm.llmapi.llm_args.RayPlacementConfig]) – prototype Placement config for RayGPUWorker. Only used with AsyncLLM and orchestrator_type=’ray’. Defaults to None.
ray_worker_nsight_options (Optional[dict[str, str]]) – prototype Nsight options. Defaults to None.
sleep_config (Optional[tensorrt_llm.llmapi.llm_args.SleepConfig]) – prototype Configuration for the LLM sleep feature. Sleep feature requires extra setup that may slow down model loading. Only enable it if you intend to use this feature. Defaults to None.
reorder_policy_config (Optional[tensorrt_llm.llmapi.llm_args.ReorderRequestPolicyConfig]) – prototype The request reordering policy to use. Defaults to None.
enable_resource_governor (bool) – prototype Enable the resource governor for runtime cache management operations such as KV cache truncation. This adds a per-iteration broadcast collective. Defaults to False.
use_cute_dsl_blockscaling_mm (bool) – prototype If true, use CuTe DSL fp8 blockscaling mm implementation. Defaults to False.
use_cute_dsl_blockscaling_bmm (bool) – prototype If true, use CuTe DSL fp8 blockscaling bmm implementation. Defaults to False.
use_cute_dsl_bf16_bmm (bool) – prototype If true, use CuTe DSL bf16 persistent GEMM for BMM on Blackwell. Defaults to False.
use_cute_dsl_bf16_gemm (bool) – prototype If true, use CuTe DSL bf16 persistent GEMM for Linear layers on Blackwell. Defaults to False.
max_stats_len (int) – prototype The max number of performance statistic entries. Set to -1 to keep all entries. Set to 0 to use a minimum buffer size of 1. Defaults to 1000.
layer_wise_benchmarks_config (tensorrt_llm.llmapi.llm_args.LayerwiseBenchmarksConfig) – prototype Defaults to None.

tokenizer#

The tokenizer loaded by LLM instance, if any.

Type:: tensorrt_llm.llmapi.tokenizer.TokenizerBase, optional

llm_id#

The unique ID of the LLM instance.

Type:: str

disaggregated_params#

The disaggregated parameters of the LLM instance.

Type:: dict

startup_metrics#

The startup metrics reported by worker rank 0.

Type:: dict

__init__(

model: str | Path,

tokenizer: str | Path | TokenizerBase | PreTrainedTokenizerBase | None = None,

tokenizer_mode: Literal['auto', 'slow'] = 'auto',

skip_tokenizer_init: bool = False,

trust_remote_code: bool = False,

tensor_parallel_size: int = 1,

dtype: str = 'auto',

revision: str | None = None,

tokenizer_revision: str | None = None,

**kwargs: Any,

) → None[source]#

prototype Encode inputs using an encoder-only model (PyTorch backend only).

Only available when encode_only=True is set in the LLM constructor.

Parameters:

inputs (tensorrt_llm.inputs.data.PromptInputs, Sequence[tensorrt_llm.inputs.data.PromptInputs]) – The prompt text or token ids. It can be a single prompt or batched prompts.
add_special_tokens (bool) – Whether to add special tokens (e.g., [CLS]/[SEP]) during tokenization. Defaults to True.
batch_indexed_model_output (bool) – If specified, assume batched model output indexed by request index, as opposed to token index. Defaults to True.
copy_logits_to_host (bool) – If set, copy logits from device to host. Otherwise, return a view into the on-device logits tensor. Defaults to True.
return_raw_logits (bool) – Whether to return the raw CPU logits tensor for the whole input batch. Defaults to False.
model_kwargs (Any) – Model-specific inputs passed through to the model’s forward(). Examples: token_type_ids (BERT), inputs_embeds (reward models).

Returns:

If return_raw_logits=True, returns the raw CPU logits tensor for the whole input batch. Otherwise, returns one EncoderOutput for a single input, or a list of EncoderOutput objects for batched inputs.

Return type:

Union[tensorrt_llm.llmapi.llm.EncoderOutput, List[tensorrt_llm.llmapi.llm.EncoderOutput], torch.Tensor]

Raises:

RuntimeError – If encode_only mode is not enabled.

Generate output for the given prompts in the synchronous mode. Synchronous generation accepts either single prompt or batched prompts.

Parameters:

inputs (tensorrt_llm.inputs.data.PromptInputs, Sequence[tensorrt_llm.inputs.data.PromptInputs]) – The prompt text or token ids. It can be single prompt or batched prompts.
sampling_params (tensorrt_llm.sampling_params.SamplingParams, List[tensorrt_llm.sampling_params.SamplingParams], optional) – The sampling params for the generation. Defaults to None. A default one will be used if not provided.
use_tqdm (bool) – Whether to use tqdm to display the progress bar. Defaults to True.
lora_request (tensorrt_llm.executor.request.LoRARequest, Sequence[tensorrt_llm.executor.request.LoRARequest], optional) – LoRA request to use for generation, if any. Defaults to None.
prompt_adapter_request (tensorrt_llm.executor.request.PromptAdapterRequest, Sequence[tensorrt_llm.executor.request.PromptAdapterRequest], optional) – Prompt Adapter request to use for generation, if any. Defaults to None.
kv_cache_retention_config (tensorrt_llm.bindings.executor.KvCacheRetentionConfig, Sequence[tensorrt_llm.bindings.executor.KvCacheRetentionConfig], optional) – Configuration for the request’s retention in the KV Cache. Defaults to None.
disaggregated_params (tensorrt_llm.disaggregated_params.DisaggregatedParams, Sequence[tensorrt_llm.disaggregated_params.DisaggregatedParams], optional) – Disaggregated parameters. Defaults to None.
scheduling_params (tensorrt_llm.scheduling_params.SchedulingParams, List[tensorrt_llm.scheduling_params.SchedulingParams], optional) – Scheduling parameters. Defaults to None.
conversation_params (tensorrt_llm.conversation_params.ConversationParams, List[tensorrt_llm.conversation_params.ConversationParams], optional) – Conversation parameters. Defaults to None.
cache_salt (str, Sequence[str], optional) – If specified, KV cache will be salted with the provided string to limit the kv cache reuse to the requests with the same string. Defaults to None.
priority (float, List[float]) – The scheduling priority for the request(s), in the range [0, 1]. Higher values indicate higher priority. Defaults to 0.5.

Returns:

The output data of the completion request to the LLM.

Return type:

Union[tensorrt_llm.llmapi.llm.RequestOutput, List[tensorrt_llm.llmapi.llm.RequestOutput]]

generate_async( inputs: str | List[int] | TextPrompt | TokensPrompt | PreprocessedInputs, sampling_params: SamplingParams | None = None, lora_request: LoRARequest | None = None, prompt_adapter_request: PromptAdapterRequest | None = None, streaming: bool = False, kv_cache_retention_config: KvCacheRetentionConfig | None = None, disaggregated_params: DisaggregatedParams | None = None, trace_headers: Mapping[str, str] | None = None, _postproc_params: PostprocParams | None = None, scheduling_params: SchedulingParams | None = None, conversation_params: ConversationParams | None = None, cache_salt: str | None = None, priority: float = 0.5, ) → RequestOutput#

Generate output for the given prompt in the asynchronous mode. Asynchronous generation accepts single prompt only.

Parameters:

inputs (Union[tensorrt_llm.inputs.data.PromptInputs, tensorrt_llm.llmapi.llm.PreprocessedInputs]) – The prompt text or token ids, or a PreprocessedInputs returned by preprocess. If the latter, preprocessing will be skipped by this method.
sampling_params (tensorrt_llm.sampling_params.SamplingParams, optional) – The sampling params for the generation. Defaults to None. A default one will be used if not provided.
lora_request (tensorrt_llm.executor.request.LoRARequest, optional) – LoRA request to use for generation, if any. Defaults to None.
prompt_adapter_request (tensorrt_llm.executor.request.PromptAdapterRequest, optional) – Prompt Adapter request to use for generation, if any. Defaults to None.
streaming (bool) – Whether to use the streaming mode for the generation. Defaults to False.
kv_cache_retention_config (tensorrt_llm.bindings.executor.KvCacheRetentionConfig, optional) – Configuration for the request’s retention in the KV Cache. Defaults to None.
disaggregated_params (tensorrt_llm.disaggregated_params.DisaggregatedParams, optional) – Disaggregated parameters. Defaults to None.
trace_headers (Mapping[str, str], optional) – Trace headers. Defaults to None.
scheduling_params (tensorrt_llm.scheduling_params.SchedulingParams, optional) – Scheduling parameters. Defaults to None.
conversation_params (tensorrt_llm.conversation_params.ConversationParams, optional) – Conversation parameters. Defaults to None.
cache_salt (str, optional) – If specified, KV cache will be salted with the provided string to limit the kv cache reuse to the requests with the same string. Defaults to None.
priority (float) – The scheduling priority for the request, in the range [0, 1]. Higher values indicate higher priority. Defaults to 0.5.

Returns:

The output data of the completion request to the LLM.

Return type:

tensorrt_llm.llmapi.llm.RequestOutput

get_data_transceiver_state() → bytes#

prototype Get the serialized DataTransceiverState for arbitrary KV cache transfer.

Returns:: Serialized DataTransceiverState, or empty bytes if no transceiver is configured.
Return type:: bytes

get_kv_cache_capacity() → dict#

beta Get the runtime’s static primary/GPU KV cache capacity.

Raises:

RuntimeError – If called when encode_only=True.

Returns:

KV cache capacity. The returned capacity covers the primary: GPU KV cache pool only; CPU/host offload capacity is not included. e.g., {“maxNumBlocks”: …, “tokensPerBlock”: …, “maxNumTokens”: …}

Return type:

dict

get_kv_cache_events( timeout: float | None = 2, ) → List[dict]#

beta Get iteration KV events from the runtime.

KV events are used to track changes and operations within the KV Cache. Types of events:

KVCacheCreatedData: Indicates the creation of cache blocks.
KVCacheStoredData: Represents a sequence of stored blocks.
KVCacheRemovedData: Contains the hashes of blocks that are being removed from the cache.
KVCacheUpdatedData: Captures updates to existing cache blocks.

To enable KV events:

set event_buffer_max_size to a positive integer in the KvCacheConfig.
set enable_block_reuse to True in the KvCacheConfig.

Parameters:: timeout (float, optional) – Max wait time in seconds when retrieving events from queue. Defaults to 2.
Returns:: A list of runtime events as dict.
Return type:: List[dict]

get_kv_cache_events_async( timeout: float | None = 2, ) → IterationResult#

beta Get iteration KV events from the runtime.

KV events are used to track changes and operations within the KV Cache. Types of events:

KVCacheCreatedData: Indicates the creation of cache blocks.
KVCacheStoredData: Represents a sequence of stored blocks.
KVCacheRemovedData: Contains the hashes of blocks that are being removed from the cache.
KVCacheUpdatedData: Captures updates to existing cache blocks.

To enable KV events:

set event_buffer_max_size to a positive integer in the KvCacheConfig.
set enable_block_reuse to True in the KvCacheConfig.

Parameters:: timeout (float, optional) – Max wait time in seconds when retrieving events from queue. Defaults to 2.
Returns:: An async iterable object containing runtime events.
Return type:: tensorrt_llm.executor.result.IterationResult

get_stats(timeout: float | None = 2) → List[dict]#

beta Get iteration statistics from the runtime. To collect statistics, call this function after prompts have been submitted with LLM().generate().

Parameters:

timeout (float, optional) – Max wait time in seconds when retrieving stats from queue. Defaults to 2.

Returns:

A list of runtime stats as dicts.: e.g., [{“cpuMemUsage”: …, “iter”: 0, …}, {“cpuMemUsage”: …, “iter”: 1, …}]

Return type:

List[dict]

get_stats_async( timeout: float | None = 2, ) → IterationResult#

beta Get iteration statistics from the runtime. To collect statistics, you can call this function in an async coroutine or the /metrics endpoint (if you’re using trtllm-serve) after prompts have been submitted.

Parameters:: timeout (float, optional) – Max wait time in seconds when retrieving stats from queue. Defaults to 2.
Returns:: An async iterable object containing runtime stats.
Return type:: tensorrt_llm.executor.result.IterationResult

prototype Preprocess raw prompts into token IDs and multimodal params.

Parameters:

inputs (tensorrt_llm.inputs.data.PromptInputs) – The prompt text or token ids; it must be single prompt.
sampling_params (tensorrt_llm.sampling_params.SamplingParams, optional) – The sampling params for the generation. Defaults to None. A default one will be used if not provided.
disaggregated_params (tensorrt_llm.disaggregated_params.DisaggregatedParams, optional) – Disaggregated parameters. Defaults to None.

Returns:

A preprocessed-inputs object that can be: passed directly to generate_async() as inputs.

Return type:

tensorrt_llm.llmapi.llm.PreprocessedInputs

shutdown() → None#: beta None

property disaggregated_params: dict#: beta None

property llm_id: str#: beta None

property startup_metrics: dict#

beta Cache and return rank-0 startup metrics.

Returns:: The cached metrics, or an empty dict when metrics retrieval fails.
Return type:: dict

property tokenizer: TokenizerBase | None#