SamplingParams#

class tensorrt_llm.llmapi.SamplingParams( *, end_id: int | None = None, pad_id: int | None = None, max_tokens: int = 32, bad: str | List[str] | None = None, bad_token_ids: List[int] | None = None, stop: str | List[str] | None = None, stop_token_ids: List[int] | None = None, include_stop_str_in_output: bool = False, embedding_bias: Tensor | None = None, logits_processor: LogitsProcessor | List[LogitsProcessor] | None = None, apply_batched_logits_processor: bool = False, n: int = 1, best_of: int | None = None, use_beam_search: bool = False, logprobs_mode: LogprobMode = LogprobMode.RAW, top_k: int | None = None, top_p: float | None = None, top_p_min: float | None = None, top_p_reset_ids: int | None = None, top_p_decay: float | None = None, seed: int | None = None, temperature: float | None = None, min_tokens: int | None = None, beam_search_diversity_rate: float | None = None, repetition_penalty: float | None = None, presence_penalty: float | None = None, frequency_penalty: float | None = None, prompt_ignore_length: int | None = None, length_penalty: float | None = None, early_stopping: int | None = None, no_repeat_ngram_size: int | None = None, min_p: float | None = None, beam_width_array: List[int] | None = None, logprobs: int | None = None, prompt_logprobs: int | None = None, logprobs_simple_format: bool = False, prompt_logprobs_simple_format: bool = False, return_context_logits: bool = False, return_generation_logits: bool = False, exclude_input_from_output: bool = True, return_encoder_output: bool = False, return_perf_metrics: bool = False, additional_model_outputs: List[str] | None = None, _context_logits_auto_enabled: bool = False, _generation_logits_auto_enabled: bool = False, _return_log_probs: bool = False, guided_decoding: GuidedDecodingParams | None = None, thinking_token_budget: int | None = None, ignore_eos: bool = False, detokenize: bool = True, add_special_tokens: bool = True, truncate_prompt_tokens: int | None = None, skip_special_tokens: bool = True, spaces_between_special_tokens: bool = True, )[source]#

Bases: object

Sampling parameters for text generation.

Usage Examples:

use_beam_search is False:

best_of is None: (top-p/top-k) sampling n responses and return n generations

best_of is not None: (top-p/top-k) sampling best_of responses and return n generations (best_of >= n must hold)

use_beam_search is True:

best_of is None: beam search with beam width of n, return n generations

best_of is not None: beam search with beam width of best_of, return n generations (best_of >= n must hold)

Parameters:

end_id (int, optional) – The end token id. Defaults to None.
pad_id (int, optional) – The pad token id. Defaults to None.
max_tokens (int) – The maximum number of tokens to generate. Defaults to 32.
bad (str, List[str], optional) – A string or a list of strings that redirect the generation when they are generated, so that the bad strings are excluded from the returned output. Defaults to None.
bad_token_ids (List[int], optional) – A list of token ids that redirect the generation when they are generated, so that the bad ids are excluded from the returned output. Defaults to None.
stop (str, List[str], optional) – A string or a list of strings that stop the generation when they are generated. The returned output will not contain the stop strings unless include_stop_str_in_output is True. Defaults to None.
stop_token_ids (List[int], optional) – A list of token ids that stop the generation when they are generated. Defaults to None.
include_stop_str_in_output (bool) – Whether to include the stop strings in output text. Defaults to False.
embedding_bias (torch.Tensor, optional) – The embedding bias tensor. Expected type is kFP32 and shape is [vocab_size]. Defaults to None.
logits_processor (tensorrt_llm.sampling_params.LogitsProcessor, List[tensorrt_llm.sampling_params.LogitsProcessor], optional) – The logits postprocessor callback(s). Defaults to None. If a list, each processor is applied in order during generation (supported in PyTorch backend only).
apply_batched_logits_processor (bool) – Whether to apply batched logits postprocessor callback. Defaults to False. The BatchedLogitsProcessor class is recommended for callback creation. The callback must be provided when initializing LLM.
n (int) – Number of sequences to generate. Defaults to 1.
best_of (int, optional) – Number of sequences to consider for best output. Defaults to None.
use_beam_search (bool) – Whether to use beam search. Defaults to False.
top_k (int, optional) – Controls number of logits to sample from. Can assume non-negative values, where 0 means ‘all logits’. Defaults to None. The value None is treated as “not specified” in the following. If neither temperature, top_p, nor top_k are specified, sampling is greedy. If temperature > 0 and/or top_p < 1 are specified, sampling will proceed accordingly and top_k will default to top_k = 0. Setting top_k = 1 results in greedy sampling.
top_p (float, optional) – Controls the top-P probability to sample from. Can have values between 0 and 1. Defaults to None. The value None is treated as “not specified” in the following. If neither temperature, top_p, nor top_k are specified, sampling is greedy. If temperature > 0 and/or top_k > 1 are specified, sampling will proceed accordingly and top_p will default to top_p = 1. Setting top_p = 0 should result in greedy sampling, but is currently disallowed in the backend.
top_p_min (float, optional) – Controls decay in the top-P algorithm. topPMin is lower-bound. Must be in (0, 1]; invalid values are rejected. None means using C++ runtime default 1.e-6. Defaults to None.
top_p_reset_ids (int, optional) – Controls decay in the top-P algorithm. The token id which, when sampled, resets the decayed top-P to its initial value. Must be >= 0; invalid values are rejected. None means using C++ runtime default -1 (which never matches a token). Defaults to None.
top_p_decay (float, optional) – Controls decay in the top-P algorithm. The decay value. Must be in (0, 1]; invalid values are rejected. None means using C++ runtime default 1.f. Defaults to None.
seed (int, optional) – Controls the random seed used by the random number generator in sampling. None means using C++ runtime default 0. Defaults to None.
temperature (float, optional) – Controls the modulation of logits when sampling new tokens. It can have values >= 0.f. Defaults to None. The value None is treated as “not specified” in the following. If neither temperature, top_p, nor top_k are specified, sampling is greedy. If top_p < 1 and/or top_k > 1 are specified, sampling will proceed accordingly and temperature will default to temperature = 1. Setting temperature = 0 results in greedy sampling.
min_tokens (int, optional) – Lower bound on the number of tokens to generate. Values < 1 have no effect. None means using C++ runtime default 1. Defaults to None.
beam_search_diversity_rate (float, optional) – Encourages beams to diverge from each other by adding diversity_rate * source_beam_index to each candidate’s ranking score during beam expansion, boosting candidates that expand from lower-ranked beams. Here source_beam_index is the rank of the beam a candidate expands from among the current step’s input beams, ordered by cumulative log-probability (0 for the strongest beam). None means using C++ runtime default 0.f (disabled). Defaults to None.
repetition_penalty (float, optional) – Used to penalize tokens based on how often they appear in the sequence. It can have any value > 0.f. Values < 1.f encourages repetition, values > 1.f discourages it. None means using C++ runtime default 1.f. Defaults to None.
presence_penalty (float, optional) – Used to penalize tokens already present in the sequence (irrespective of the number of appearances). It can have any values. Values < 0.f encourage repetition, values > 0.f discourage it. None means using C++ runtime default 0.f. Defaults to None.
frequency_penalty (float, optional) – Used to penalize tokens already present in the sequence (dependent on the number of appearances). It can have any values. Values < 0.f encourage repetition, values > 0.f discourage it. None means using C++ runtime default 0.f. Defaults to None.
prompt_ignore_length (int, optional) – Controls how many tokens to ignore from the prompt for presence and frequency penalties. Values <= 0 have no effect. Values > input (prompt) length will be clamped. None means using C++ runtime default 0. Defaults to None.
length_penalty (float, optional) – Beam-search length penalty exponent. Beams are ranked by cum_log_prob / length**length_penalty, where length counts generated tokens only; the returned cumulative_logprob stays unnormalized. Must be >= 0. 0 disables the normalization. None means using C++ runtime default 0.f. Defaults to None.
early_stopping (int, optional) – Three-state, following HuggingFace. 1 stops as soon as best_of finished candidates exist. 0 and 2 are exhaustive: they keep a pool of finished candidates and continue while an unfinished beam could still outscore the worst of them, 0 bounding attainability with the beams’ current length and 2 (“never”) with max_seq_len when length_penalty > 0. Any other integer is treated as 2. None means using C++ runtime default 1. Defaults to None.
no_repeat_ngram_size (int, optional) – Forbids repeating any n-gram of this size: a token is excluded from sampling if it would recreate an n-gram that already occurs in the sequence (prompt included). None or 0 disables the restriction. Defaults to None.
min_p (float, optional) – scale the most likely token to determine the minimum token probability. None means using C++ runtime default 0.0. Defaults to None.
beam_width_array (List[int], optional) – Per-iteration beam widths for Variable-Beam-Width-Search; decoding past the end of the array holds its last entry. Must be non-decreasing – a narrowing schedule is rejected. beam_width is raised to the array’s maximum, which is the number of beams returned. Defaults to None.
logprobs (int, optional) – Number of log probabilities to return per output token. When set to 0, return only the sampled token’s log probability. When set to K>0, return top-K log probabilities + the sampled token’s log probability (last entry) if it’s not in the Top-K. Defaults to None.
logprobs_mode (LogprobMode) – The mode of log probabilities to return. Defaults to LogprobMode.RAW.
logprobs_simple_format (bool) – If True (and logprobs == 0), return generation logprobs as a flat list[float] (one logprob per generated token) instead of the default list[dict[int, Logprob]] format. Reduces per-token allocation overhead when only the sampled-token logprob is needed. Incompatible with logprobs is None or logprobs > 0 and with beam search. Defaults to False.
prompt_logprobs (int, optional) – Number of log probabilities to return per prompt token. When set to 0, return only the actual prompt token’s log probability. When set to K>0, return top-K log probabilities + the actual prompt token’s log probability (last entry) if it’s not in the Top-K. Defaults to None.
prompt_logprobs_simple_format (bool) – If True (and prompt_logprobs == 0), return prompt logprobs as a flat list[float] instead of the default list[dict[int, Logprob]] format. Incompatible with prompt_logprobs is None or prompt_logprobs > 0. Defaults to False.
return_context_logits (bool) – Controls if Result should contain the context logits. Defaults to False.
return_generation_logits (bool) – Controls if Result should contain the generation logits. Defaults to False.
exclude_input_from_output (bool) – Controls if output tokens in Result should include the input tokens. Defaults to True.
return_encoder_output (bool) – Controls if Result should contain encoder output hidden states (for encoder-only and encoder-decoder models). Defaults to False.
return_perf_metrics (bool) – Controls if Result should contain the performance metrics for this request. Defaults to False.
additional_model_outputs (List[str], optional) – The additional outputs to gather from the model. Defaults to None.
guided_decoding (tensorrt_llm.sampling_params.GuidedDecodingParams, optional) – Guided decoding params. Defaults to None.
thinking_token_budget (int, optional) – Experimental. Maximum number of tokens allowed inside a reasoning block. Set to -1 or None for unlimited. Defaults to None.
ignore_eos (bool) – Whether to ignore the EOS token and continue generating tokens after the EOS token is generated. Defaults to False.
detokenize (bool) – Whether to detokenize the output. Defaults to True.
add_special_tokens (bool) – Whether to add special tokens to the prompt. Defaults to True.
truncate_prompt_tokens (int, optional) – If set to an integer k, will use only the last k tokens from the prompt (i.e., left truncation). Defaults to None.
skip_special_tokens (bool) – Whether to skip special tokens in the output. Defaults to True.
spaces_between_special_tokens (bool) – Whether to add spaces between special tokens in the output. Defaults to True.

__init__( *, end_id: int | None = None, pad_id: int | None = None, max_tokens: int = 32, bad: str | List[str] | None = None, bad_token_ids: List[int] | None = None, stop: str | List[str] | None = None, stop_token_ids: List[int] | None = None, include_stop_str_in_output: bool = False, embedding_bias: Tensor | None = None, logits_processor: LogitsProcessor | List[LogitsProcessor] | None = None, apply_batched_logits_processor: bool = False, n: int = 1, best_of: int | None = None, use_beam_search: bool = False, logprobs_mode: LogprobMode = LogprobMode.RAW, top_k: int | None = None, top_p: float | None = None, top_p_min: float | None = None, top_p_reset_ids: int | None = None, top_p_decay: float | None = None, seed: int | None = None, temperature: float | None = None, min_tokens: int | None = None, beam_search_diversity_rate: float | None = None, repetition_penalty: float | None = None, presence_penalty: float | None = None, frequency_penalty: float | None = None, prompt_ignore_length: int | None = None, length_penalty: float | None = None, early_stopping: int | None = None, no_repeat_ngram_size: int | None = None, min_p: float | None = None, beam_width_array: List[int] | None = None, logprobs: int | None = None, prompt_logprobs: int | None = None, logprobs_simple_format: bool = False, prompt_logprobs_simple_format: bool = False, return_context_logits: bool = False, return_generation_logits: bool = False, exclude_input_from_output: bool = True, return_encoder_output: bool = False, return_perf_metrics: bool = False, additional_model_outputs: List[str] | None = None, _context_logits_auto_enabled: bool = False, _generation_logits_auto_enabled: bool = False, _return_log_probs: bool = False, guided_decoding: GuidedDecodingParams | None = None, thinking_token_budget: int | None = None, ignore_eos: bool = False, detokenize: bool = True, add_special_tokens: bool = True, truncate_prompt_tokens: int | None = None, skip_special_tokens: bool = True, spaces_between_special_tokens: bool = True, ) → None#

static params_imply_explicit_greedy( *, temperature: float | None, top_p: float | None, top_k: int | None, min_p: float | None, ) → bool[source]#

Whether the request carries an explicit greedy control.

Explicit means top_k == 1, top_p == 0.0, min_p == 1.0, or temperature == 0, as opposed to the implicit “all params unset” greedy default. min_p == 1.0 keeps only tokens whose probability equals the row maximum, so like top_p == 0.0 it collapses sampling to a single token.

Whether the parameters resolve to greedy decoding.

An explicit greedy control always wins. The implicit “all params unset” greedy default is overridden by any active sampling knob: an active top-p decay (which implies top-p sampling so the decayed runtime top-p can take effect) or a min_p in (0, 1) (which still selects among multiple tokens); callers that do not support decay may omit top_p_decay.

static params_imply_top_p_decay_active( top_p_decay: float | None, ) → bool[source]#

Whether dynamic top-p decay is active.

Active iff top_p_decay is explicitly set and < 1.0; a decay of 1.0 (the C++ default) is a no-op. Values outside (0, 1] are rejected up front (_validate and the executor::SamplingConfig constructor), so they never reach this predicate.

add_special_tokens: bool#

additional_model_outputs: List[str] | None#

apply_batched_logits_processor: bool#

bad: str | List[str] | None#

bad_token_ids: List[int] | None#

beam_search_diversity_rate: float | None#

beam_width_array: List[int] | None#

best_of: int | None#

detokenize: bool#

early_stopping: int | None#

embedding_bias: Tensor | None#

end_id: int | None#

exclude_input_from_output: bool#

frequency_penalty: float | None#

guided_decoding: GuidedDecodingParams | None#

ignore_eos: bool#

include_stop_str_in_output: bool#

length_penalty: float | None#

logits_processor: LogitsProcessor | List[LogitsProcessor] | None#

logprobs: int | None#

logprobs_mode: LogprobMode#

logprobs_simple_format: bool#

max_tokens: int#

min_p: float | None#

min_tokens: int | None#

n: int#

no_repeat_ngram_size: int | None#

pad_id: int | None#

presence_penalty: float | None#

prompt_ignore_length: int | None#

prompt_logprobs: int | None#

prompt_logprobs_simple_format: bool#

repetition_penalty: float | None#

return_context_logits: bool#

return_encoder_output: bool#

return_generation_logits: bool#

return_perf_metrics: bool#

seed: int | None#

skip_special_tokens: bool#

spaces_between_special_tokens: bool#

stop: str | List[str] | None#

stop_token_ids: List[int] | None#

temperature: float | None#

thinking_token_budget: int | None#

top_k: int | None#

top_p: float | None#

top_p_decay: float | None#

top_p_min: float | None#

top_p_reset_ids: int | None#

truncate_prompt_tokens: int | None#

use_beam_search: bool#