API Reference#

Note

Since version 1.0, we have attached a status label to LLM, LlmArgs and TorchLlmArgs Classes.

stable - The item is stable and will keep consistent.
prototype - The item is a prototype and is subject to change.
beta - The item is in beta and approaching stability.
deprecated - The item is deprecated and will be removed in a future release.

This page provides a searchable index of API names and parameters. Open an entry’s full reference for complete member and inheritance documentation.

class tensorrt_llm.llmapi.LLM(

model,

tokenizer,

tokenizer_mode,

skip_tokenizer_init,

trust_remote_code,

tensor_parallel_size,

dtype,

revision,

tokenizer_revision,

**kwargs,

)[source]: View the full LLM reference

class tensorrt_llm.llmapi.AsyncLLM(

placement_groups,

placement_bundle_indices,

per_worker_gpu_share,

*args,

**kwargs,

)[source]: View the full AsyncLLM reference

class tensorrt_llm.llmapi.MultimodalEncoder(

model,

trust_remote_code,

tensor_parallel_size,

dtype,

**kwargs,

)[source]: View the full MultimodalEncoder reference

class tensorrt_llm.llmapi.CompletionOutput( index, text, token_ids, cumulative_logprob, logprobs, prompt_logprobs, finish_reason, stop_reason, generation_logits, additional_context_outputs, additional_generation_outputs, disaggregated_params, request_perf_metrics, _postprocess_result, )[source]: View the full CompletionOutput reference

class tensorrt_llm.llmapi.RequestOutput[source]: View the full RequestOutput reference

class tensorrt_llm.llmapi.GuidedDecodingParams( *, json, regex, grammar, json_object, structural_tag, )[source]: View the full GuidedDecodingParams reference

class tensorrt_llm.llmapi.SamplingParams( *, end_id, pad_id, max_tokens, bad, bad_token_ids, stop, stop_token_ids, include_stop_str_in_output, embedding_bias, logits_processor, apply_batched_logits_processor, n, best_of, use_beam_search, logprobs_mode, top_k, top_p, top_p_min, top_p_reset_ids, top_p_decay, seed, temperature, min_tokens, beam_search_diversity_rate, repetition_penalty, presence_penalty, frequency_penalty, prompt_ignore_length, length_penalty, early_stopping, no_repeat_ngram_size, min_p, beam_width_array, logprobs, prompt_logprobs, logprobs_simple_format, prompt_logprobs_simple_format, return_context_logits, return_generation_logits, exclude_input_from_output, return_encoder_output, return_perf_metrics, additional_model_outputs, _context_logits_auto_enabled, _generation_logits_auto_enabled, _return_log_probs, guided_decoding, thinking_token_budget, ignore_eos, detokenize, add_special_tokens, truncate_prompt_tokens, skip_special_tokens, spaces_between_special_tokens, )[source]: View the full SamplingParams reference

class tensorrt_llm.llmapi.DisaggregatedParams( *, request_type, first_gen_tokens, first_gen_log_probs, first_gen_logits, ctx_request_id, opaque_state, draft_tokens, disagg_request_id, ctx_dp_rank, ctx_info_endpoint, schedule_style, ctx_usage, multimodal_embedding_handles, multimodal_hashes, mrope_position_ids_handle, mrope_position_deltas_handle, )[source]: View the full DisaggregatedParams reference

class tensorrt_llm.llmapi.ConversationParams(*, conversation_id)[source]: View the full ConversationParams reference

class tensorrt_llm.llmapi.DisaggScheduleStyle(*values)[source]: View the full DisaggScheduleStyle reference

class tensorrt_llm.llmapi.BlockReuseConfig(*, policy, max_num_turns)[source]: View the full BlockReuseConfig reference

class tensorrt_llm.llmapi.KvCacheConfig( *, enable_block_reuse, max_tokens, max_attention_window, sink_token_length, free_gpu_memory_fraction, host_cache_size, disk_cache_size, disk_cache_path, cross_kv_cache_fraction, secondary_offload_min_priority, event_buffer_max_size, attention_dp_events_gather_period_ms, enable_partial_reuse, copy_on_partial_reuse, use_uvm, max_gpu_total_bytes, iteration_stats_interval, dtype, mamba_ssm_cache_dtype, mamba_ssm_stochastic_rounding, mamba_ssm_philox_rounds, tokens_per_block, mamba_state_cache_interval, mamba_state_config, use_kv_cache_manager_v2, enable_swa_scratch_reuse, kv_cache_event_hash_algo, max_util_for_resume, enable_kv_pool_rebalance, disk_prefetch_num_reqs, fp8_context_mla_kv_len_cap, pool_ratio, avg_seq_len, block_reuse_config, )[source]: View the full KvCacheConfig reference

class tensorrt_llm.llmapi.MambaStateConfig( *, periodic_snapshot_interval, additional_snapshot_offsets_from_start, additional_snapshot_offsets_from_end, enable_branch_snapshot, )[source]: View the full MambaStateConfig reference

class tensorrt_llm.llmapi.KvCacheRetentionConfig(*args, **kwargs): View the full KvCacheRetentionConfig reference

tensorrt_llm.llmapi.CudaGraphConfig: alias of DecodeCudaGraphConfig

class tensorrt_llm.llmapi.DecodeCudaGraphConfig( *, batch_sizes, max_batch_size, enable_padding, mode, )[source]: View the full DecodeCudaGraphConfig reference

class tensorrt_llm.llmapi.EncodeCudaGraphConfig( *, batch_sizes, max_batch_size, enable_padding, mode, num_tokens, max_num_token, seq_lens, max_seq_len, )[source]: View the full EncodeCudaGraphConfig reference

class tensorrt_llm.llmapi.MoeConfig( *, backend, max_num_tokens, load_balancer, disable_finalize_fusion, use_low_precision_moe_combine, )[source]: View the full MoeConfig reference

class tensorrt_llm.llmapi.EagleDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, eagle_choices, greedy_sampling, posterior_threshold, use_dynamic_tree, dynamic_tree_max_topK, num_eagle_layers, max_non_leaves_per_layer, eagle3_one_model, eagle3_layers_to_capture, eagle3_model_arch, )[source]: View the full EagleDecodingConfig reference

class tensorrt_llm.llmapi.Eagle3DecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, eagle_choices, greedy_sampling, posterior_threshold, use_dynamic_tree, dynamic_tree_max_topK, num_eagle_layers, max_non_leaves_per_layer, eagle3_one_model, eagle3_layers_to_capture, eagle3_model_arch, sa_config, )[source]: View the full Eagle3DecodingConfig reference

class tensorrt_llm.llmapi.MTPDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, use_relaxed_acceptance_for_thinking, relaxed_topk, relaxed_delta, use_mtp_vanilla, mtp_eagle_one_model, use_dynamic_tree, dynamic_tree_max_topK, sa_config, num_nextn_predict_layers, begin_thinking_phase_token, end_thinking_phase_token, )[source]: View the full MTPDecodingConfig reference

class tensorrt_llm.llmapi.SchedulerConfig( *, capacity_scheduler_policy, context_chunking_policy, dynamic_batch_config, waiting_queue_policy, use_python_scheduler, enable_prefix_aware_scheduling, )[source]: View the full SchedulerConfig reference

class tensorrt_llm.llmapi.CapacitySchedulerPolicy(*values)[source]: View the full CapacitySchedulerPolicy reference

class tensorrt_llm.llmapi.QuantConfig( *, quant_algo, kv_cache_quant_algo, group_size, smoothquant_val, clamp_val, use_meta_recipe, has_zero_point, pre_quant_scale, exclude_modules, mamba_ssm_cache_dtype, mamba_ssm_stochastic_rounding, mamba_ssm_philox_rounds, )[source]: View the full QuantConfig reference

class tensorrt_llm.llmapi.QuantAlgo(*values)[source]: View the full QuantAlgo reference

class tensorrt_llm.llmapi.CalibConfig( *, device, calib_dataset, calib_batches, calib_batch_size, calib_max_seq_length, random_seed, tokenizer_max_seq_length, )[source]: View the full CalibConfig reference

class tensorrt_llm.llmapi.RequestError[source]: View the full RequestError reference

class tensorrt_llm.llmapi.MpiCommSession(comm, n_workers)[source]: View the full MpiCommSession reference

class tensorrt_llm.llmapi.ExtendedRuntimePerfKnobConfig( *, multi_block_mode, enable_context_fmha_fp32_acc, cuda_graph_mode, cuda_graph_cache_size, )[source]: View the full ExtendedRuntimePerfKnobConfig reference

class tensorrt_llm.llmapi.BatchingType(*values)[source]: View the full BatchingType reference

class tensorrt_llm.llmapi.ContextChunkingPolicy(*values)[source]: View the full ContextChunkingPolicy reference

class tensorrt_llm.llmapi.DynamicBatchConfig( *, enable_batch_size_tuning, enable_max_num_tokens_tuning, dynamic_batch_moving_average_window, )[source]: View the full DynamicBatchConfig reference

class tensorrt_llm.llmapi.CacheTransceiverConfig( *, backend, transceiver_runtime, max_tokens_in_buffer, kv_transfer_timeout_ms, kv_transfer_sender_future_timeout_ms, kv_transfer_poll_interval_ms, kv_cache_bounce_size_mb, enable_pipelined_transfer, )[source]: View the full CacheTransceiverConfig reference

class tensorrt_llm.llmapi.NGramDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, max_matching_ngram_size, is_keep_all, is_use_oldest, is_public_pool, )[source]: View the full NGramDecodingConfig reference

class tensorrt_llm.llmapi.PARDDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, mask_token_id, decoding_type, sa_config, )[source]: View the full PARDDecodingConfig reference

class tensorrt_llm.llmapi.DFlashDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, mask_token_id, target_layer_ids, decoding_type, attention_backend, )[source]: View the full DFlashDecodingConfig reference

class tensorrt_llm.llmapi.DSparkDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, mask_token_id, target_layer_ids, block_size, markov_rank, markov_head_type, decoding_type, attention_backend, )[source]: View the full DSparkDecodingConfig reference

class tensorrt_llm.llmapi.SADecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, max_matching_ngram_size, enable_global_pool, global_pool_size, )[source]: View the full SADecodingConfig reference

class tensorrt_llm.llmapi.SAEnhancerConfig(*, threshold, enable_global_pool)[source]: View the full SAEnhancerConfig reference

class tensorrt_llm.llmapi.UserProvidedDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, drafter, resource_manager, )[source]: View the full UserProvidedDecodingConfig reference

class tensorrt_llm.llmapi.TorchCompileConfig( *, enable_fullgraph, enable_inductor, enable_piecewise_cuda_graph, capture_num_tokens, enable_userbuffers, max_num_streams, )[source]: View the full TorchCompileConfig reference

class tensorrt_llm.llmapi.DraftTargetDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, )[source]: View the full DraftTargetDecodingConfig reference

tensorrt_llm.llmapi.LlmArgs: alias of TorchLlmArgs

class tensorrt_llm.llmapi.TorchLlmArgs( *, model, tokenizer, tokenizer_mode, custom_tokenizer, post_processor_hook, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, revision, tokenizer_revision, model_kwargs, pipeline_parallel_size, context_parallel_size, gpus_per_node, moe_cluster_parallel_size, moe_tensor_parallel_size, moe_expert_parallel_size, enable_attention_dp, enable_lm_head_tp_in_adp, pp_partition, cp_config, load_format, enable_lora, lora_config, kv_cache_config, enable_chunked_prefill, guided_decoding_backend, batched_logits_processor, iter_stats_max_iterations, request_stats_max_iterations, peft_cache_config, scheduler_config, cache_transceiver_config, sparse_attention_config, kv_cache_compression_config, speculative_config, max_batch_size, max_input_len, max_seq_len, max_beam_width, max_num_tokens, gather_generation_logits, num_postprocess_workers, postprocess_tokenizer_dir, num_serve_frontends, reasoning_parser, decoding_config, _mpi_session, otlp_traces_endpoint, backend, return_perf_metrics, perf_metrics_output_dir, perf_metrics_max_requests, prometheus_metrics_config, enable_energy_metrics, orchestrator_type, env_overrides, telemetry_config, generation_config, garbage_collection_gen0_threshold, cuda_graph_config, encoder_cuda_graph_config, enable_encoder_decoder_mixed_cuda_graph, multimodal_config, attention_dp_config, disable_overlap_scheduler, moe_config, nvfp4_gemm_config, dwdp_config, encoder_max_batch_size, encoder_max_num_tokens, attn_backend, enable_mla_skip_correction, mla_skip_correction_threshold, sampler_force_async_worker, enable_speculative_beam_history_d2h, enable_early_first_token_response, enable_low_latency_host_dispatch, enable_iter_perf_stats, enable_iter_req_stats, print_iter_log, batch_wait_timeout_ms, batch_wait_timeout_iters, batch_wait_max_tokens_ratio, torch_compile_config, prefill_cuda_graph_backend, prefill_capture_num_tokens, enable_autotuner, use_fine_grained_sync, enable_layerwise_nvtx_marker, enable_min_latency, stream_interval, force_dynamic_quantization, allreduce_strategy, checkpoint_loader, checkpoint_format, checkpoint_io_policy, mx_config, gms_config, kv_connector_config, mm_encoder_only, disable_mm_encoder, encode_only, ray_worker_extension_cls, ray_placement_config, ray_worker_nsight_options, sleep_config, reorder_policy_config, enable_resource_governor, use_cute_dsl_blockscaling_mm, use_cute_dsl_blockscaling_bmm, use_cute_dsl_bf16_bmm, use_cute_dsl_bf16_gemm, max_stats_len, layer_wise_benchmarks_config, )[source]: View the full TorchLlmArgs reference

class tensorrt_llm.llmapi.AutoDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, )[source]: View the full AutoDecodingConfig reference

class tensorrt_llm.llmapi.AttentionDpConfig( *, enable_balance, timeout_iters, batching_wait_iters, enable_kv_cache_aware_routing, kv_cache_routing_load_balance_weight, kv_cache_routing_match_rate_threshold, kv_cache_routing_fair_share_multiplier, kv_cache_routing_cold_start_warmup, kv_cache_routing_account_for_in_transfer, kv_cache_routing_conversation_affinity, kv_cache_routing_max_sessions, kv_cache_routing_new_conv_placement, )[source]: View the full AttentionDpConfig reference

class tensorrt_llm.llmapi.LoRARequest(lora_name, lora_int_id, lora_path, lora_ckpt_source)[source]: View the full LoRARequest reference

class tensorrt_llm.llmapi.SaveHiddenStatesDecodingConfig( *, max_draft_len, max_total_draft_tokens, speculative_model, max_concurrency, draft_len_schedule, load_format, acceptance_rate_window_size, acceptance_rate_threshold, use_rejection_sampling, allow_advanced_sampling, advanced_sampling_mode, enable_penalty, decoding_type, output_directory, write_interval, file_prefix, eagle3_layers_to_capture, eagle_choices, )[source]: View the full SaveHiddenStatesDecodingConfig reference

class tensorrt_llm.llmapi.RocketSparseAttentionConfig( *, algorithm, seq_len_threshold, window_size, kernel_size, topr, topk, prompt_budget, page_size, kt_cache_dtype, )[source]: View the full RocketSparseAttentionConfig reference

class tensorrt_llm.llmapi.QSASparseAttentionConfig(*, algorithm, seq_len_threshold)[source]: View the full QSASparseAttentionConfig reference

class tensorrt_llm.llmapi.ReorderRequestPolicyConfig(*, policy_name, policy_args)[source]: View the full ReorderRequestPolicyConfig reference

class tensorrt_llm.llmapi.DeepSeekSparseAttentionConfig( *, algorithm, seq_len_threshold, index_n_heads, index_head_dim, index_topk, indexer_max_chunk_size, skip_indexer_for_short_seqs, use_cute_dsl_topk, use_cute_dsl_paged_mqa_logits, q_split_threshold, indexer_rope_interleave, enable_heuristic_topk, indexer_k_dtype, index_share_for_mtp_iteration, )[source]: View the full DeepSeekSparseAttentionConfig reference

class tensorrt_llm.llmapi.DeepSeekV4SparseAttentionConfig( *, algorithm, seq_len_threshold, index_n_heads, index_head_dim, index_topk, indexer_max_chunk_size, skip_indexer_for_short_seqs, use_cute_dsl_topk, use_cute_dsl_paged_mqa_logits, q_split_threshold, indexer_rope_interleave, enable_heuristic_topk, indexer_k_dtype, index_share_for_mtp_iteration, compress_ratios, window_size, )[source]: View the full DeepSeekV4SparseAttentionConfig reference

class tensorrt_llm.llmapi.MiniMaxM3SparseAttentionConfig( *, algorithm, sparse_num_index_heads, sparse_index_dim, sparse_block_size, sparse_topk_blocks, sparse_init_blocks, sparse_local_blocks, sparse_score_type, sparse_disable_index_value, indexer_kv_dtype, num_attention_heads, num_key_value_heads, implementation, )[source]: View the full MiniMaxM3SparseAttentionConfig reference

class tensorrt_llm.llmapi.SchedulingParams( *, attention_dp_rank, attention_dp_relax, agent_hierarchy, )[source]: View the full SchedulingParams reference

class tensorrt_llm.llmapi.SkipSoftmaxAttentionConfig( *, algorithm, threshold_scale_factor, target_sparsity, )[source]: View the full SkipSoftmaxAttentionConfig reference

class tensorrt_llm.llmapi.ColdPageQuantizationCompressionConfig( *, algorithm, quant, scale_checkpoint_path, )[source]: View the full ColdPageQuantizationCompressionConfig reference

class tensorrt_llm.llmapi.TriAttentionKvCacheCompressionConfig( *, algorithm, eviction_mode, normalize_scores, budget, beta, calibration_path, )[source]: View the full TriAttentionKvCacheCompressionConfig reference

class tensorrt_llm.llmapi.PrometheusMetricsConfig( *, e2e_request_latency_buckets, time_to_first_token_buckets, time_per_output_token_buckets, request_queue_time_buckets, request_prefill_time_buckets, request_decode_time_buckets, request_inference_time_buckets, )[source]: View the full PrometheusMetricsConfig reference

class tensorrt_llm.llmapi.PrefillCudaGraphBackend(*values)[source]: View the full PrefillCudaGraphBackend reference

class tensorrt_llm.llmapi.ThinkingBudgetLogitsProcessor( thinking_token_budget, reasoning_start_token_ids, reasoning_end_token_ids, )[source]: View the full ThinkingBudgetLogitsProcessor reference

class tensorrt_llm.llmapi.add_thinking_budget_logits_processor( sampling_params, *, reasoning_parser, tokenizer, chat_template_kwargs, )[source]: View the full add_thinking_budget_logits_processor reference

class tensorrt_llm.llmapi.MultimodalConfig( *, encoder_cuda_graph, encoder_side_stream_max_ahead, encoder_cache_max_bytes, encoder_scheduling_policy, video_pruning_rate, )[source]: View the full MultimodalConfig reference