Sampling#
-
struct RowSamplingParams#
Sampling parameters of one logits row, as a request carries them: the part of SamplingParams that does not depend on the launch. A plain value, so the runtime stores one per slot and rows that compare equal share a launch. topK <= 0 disables top-k.
Subclassed by trt_edgellm::SamplingParams
-
struct SamplingParams : public trt_edgellm::RowSamplingParams#
Parameters of one sampling launch: the rows’ shared RowSamplingParams plus the launch geometry.
Public Functions
- inline SamplingParams(
- int32_t batchSize_,
- int32_t vocabSize_,
- float temperature_ = 1.0f,
- int32_t topK_ = 0,
- float topP_ = 1.0f
Constructor with default values.
With top-k disabled and top-p at 1.0 (a temperature alone) the row samples from its whole temperature-scaled distribution through the top-p path, which retains everything at p = 1.0.
- Parameters:
batchSize_ – Number of samples in the batch
vocabSize_ – Size of the vocabulary
temperature_ – Temperature parameter (default: 1.0f)
topK_ – Top-K parameter (default: 0, disabled)
topP_ – Top-P parameter (default: 1.0f, disabled)
- Throws:
std::invalid_argument – if temperature is negative
- inline SamplingParams(
- int32_t batchSize_,
- int32_t vocabSize_,
- RowSamplingParams const &row
A launch over
batchSize_rows that all takerow.
-
struct SamplingRow#
One row of a per-row sampling launch: a request’s parameters as the kernels read them, with the uniform the row draws.
makeSamplingRow folds a greedy row into top-k 1, so the top-k kernels serve it; topK == 0 sends a row to the top-p kernels, which sample its whole temperature-scaled distribution at topP == 1.
- size_t trt_edgellm::getTopKtopPSamplingWorkspaceSize(
- int32_t batchSize,
- int32_t vocabSize,
- SamplingParams const ¶ms
Get workspace size required for top-K/top-P sampling (FP32 only).
Calculates the amount of GPU memory needed for intermediate computations during the sampling operation. The workspace must be allocated before calling topKtopPSamplingFromLogits().
- Parameters:
batchSize – [in] Batch size for sampling
vocabSize – [in] Vocabulary size
params – [in] Sampling parameters
- Throws:
std::runtime_error – if topK and topP are both not set
- Returns:
Required workspace size in bytes
- size_t trt_edgellm::getTopPProbabilitiesWorkspaceSize(
- int32_t rows,
- int32_t vocabSize
Return the workspace required by topPProbabilitiesFromLogits().
- size_t trt_edgellm::getSelectAllTopKWorkspaceSize(
- int32_t batchSize,
- int32_t vocabSize,
- int32_t topK
Get workspace size required for selectAllTopK operation (FP32 only).
Calculates the amount of GPU memory needed for intermediate computations during the top-K selection operation. The workspace must be allocated before calling selectAllTopK().
- Parameters:
batchSize – [in] Batch size for selection
vocabSize – [in] Vocabulary size
topK – [in] Number of top elements to select
- Returns:
Required workspace size in bytes
- size_t trt_edgellm::getRowSamplingWorkspaceSize(
- int32_t maxRows,
- int32_t vocabSize
Workspace bytes for sampleRows over at most
maxRowsrows: the top-k region at kMaxSamplingTopK and the top-p region, both resident because one launch may use both.
- void trt_edgellm::applyLogitBias(
- rt::Tensor &logits,
- rt::Tensor const &tokenIds,
- rt::Tensor const &biasValues,
- rt::Tensor const &offsets,
- cudaStream_t stream
Apply sparse per-batch logit biases in place before sampling.
Biases are represented in CSR-style flattened buffers. For each batch row
b, entries in[offsets[b], offsets[b + 1])are added tologits[b, tokenIds[i]]. Invalid token IDs are ignored defensively; callers should still validate token IDs before invoking this helper.- Parameters:
logits – [inout] Logits tensor [GPU, Float] with shape [batch-size, vocab-size]
tokenIds – [in] Flattened biased token IDs [GPU, Int32] with shape [num-biased-tokens]
biasValues – [in] Flattened bias values [GPU, Float] with shape [num-biased-tokens]
offsets – [in] Per-batch CSR offsets [GPU, Int32] with shape [batch-size + 1]
stream – [in] CUDA stream to execute the kernel
- Throws:
std::runtime_error – If tensor validation or CUDA launch fails
- void trt_edgellm::applyLogitBiasRepeatedRows(
- rt::Tensor &logits,
- rt::Tensor const &tokenIds,
- rt::Tensor const &biasValues,
- rt::Tensor const &offsets,
- int32_t rowsPerSlot,
- cudaStream_t stream
Apply sparse per-slot additive logit biases to repeated logits rows.
This variant is used by speculative verification tensors where each active request slot owns multiple contiguous logits rows. Row
rconsumes the CSR bias entries for slotr / rowsPerSlot.- Parameters:
logits – [inout] Logits tensor [GPU, Float] with shape [batch-size * rowsPerSlot, vocab-size]
tokenIds – [in] Flattened biased token IDs [GPU, Int32] with shape [num-biased-tokens]
biasValues – [in] Flattened bias values [GPU, Float] with shape [num-biased-tokens]
offsets – [in] Per-slot CSR offsets [GPU, Int32] with shape [batch-size + 1]
rowsPerSlot – [in] Number of contiguous logits rows for each active request slot
stream – [in] CUDA stream to execute the kernel
- Throws:
std::runtime_error – If tensor validation or CUDA launch fails
- void trt_edgellm::applyTokenBitmask(
- rt::Tensor &logits,
- rt::Tensor const &bitmask,
- rt::Tensor const &rowNeedsMask,
- int32_t numRows,
- cudaStream_t stream
Zero out grammar-forbidden tokens ahead of sampling.
The bitmask is dense and bit-packed, one bit per output-vocabulary entry, with a set bit meaning allowed. Its width equals the logits width, so no vocabulary lookup is needed. Row
rof the bitmask constrains rowrof the logits. Rows are not slots: each slot ownsrowsPerSlotconsecutive rows, exactly as in applyLogitBiasRepeatedRows.- Parameters:
logits – [inout] Logits [GPU, Float] with shape [num-rows, vocab-size]
bitmask – [in] Packed mask [GPU, Int32] with shape [>= num-rows, ceil(vocab-size / 32)]
rowNeedsMask – [in] Per-row flag [GPU, Int32], shape [>= num-rows]; rows set to 0 are skipped entirely, so unconstrained requests in a mixed batch cost nothing
numRows – [in] Number of logits rows to constrain
stream – [in] CUDA stream to execute the kernel
- Throws:
std::runtime_error – If tensor validation or CUDA launch fails
- void trt_edgellm::topKtopPSamplingFromLogits(
- rt::Tensor const &logits,
- rt::Tensor &selectedIndices,
- SamplingParams const ¶ms,
- rt::Tensor &workspace,
- cudaStream_t stream,
- uint64_t randomSeed = 42,
- uint64_t randomOffset = 0,
- rt::Tensor const *rowUniforms = nullptr
Main sampling function for top-K and top-P sampling from logits.
Performs token sampling using top-K and/or top-P (nucleus) sampling strategies on the input logits. The function applies temperature scaling and returns the selected token indices for each batch element.
- Parameters:
logits – [in] Input logits tensor [GPU, Float] with shape [batch-size, vocab-size]
selectedIndices – [out] Selected token indices [GPU, Int32] with shape [batch-size, 1]
params – [in] Sampling parameters including batch size, vocab size, temperature, top-K, and top-P values
workspace – [inout] Workspace buffer [GPU, Int8] for intermediate computations
stream – [in] CUDA stream to execute the kernel
randomSeed – [in] Random seed for rows without a supplied uniform (default: 42)
randomOffset – [in] Random offset for rows without a supplied uniform; use a new value per decoding step (default: 0)
rowUniforms – [in] Optional GPU FP32 tensor [batch-size]. When supplied, each row uses this request-derived uniform instead of deriving randomness from its transient batch index.
- Throws:
std::runtime_error – If CUDA operations fail
- SamplingRow trt_edgellm::makeSamplingRow(
- RowSamplingParams const ¶ms,
- float uniform
The row the kernels run for
params, drawinguniformin [0, 1).
- void trt_edgellm::sampleRows(
- rt::Tensor const &logits,
- rt::Tensor &selectedIndices,
- SamplingRow const *hostRows,
- rt::Tensor const &deviceRows,
- int32_t rowCount,
- rt::Tensor &workspace,
- cudaStream_t stream
Sample one token per row, each row with its own parameters, in a fixed number of launches.
The top-k kernels run once over every row with topK > 0 and the top-p kernels once over the rest; rows that belong to the other path return at the top of each kernel. A row’s token is what the whole-batch launch of its parameters and uniform would produce (see topKtopPSamplingFromLogits), whatever else is in the batch and however the rows are ordered.
- Parameters:
logits – [in] [GPU, Float] with shape [row-count, vocab-size]
selectedIndices – [out] [GPU, Int32] with shape [row-count, 1]
hostRows – [in] The
rowCountrows, read on the host to plan the launches: every topK in [0, kMaxSamplingTopK] and every temperature non-negativedeviceRows – [in] [GPU, Int8] holding the same rows as
rowCountSamplingRow, copied onstreamworkspace – [inout] [GPU, Int8] sized by getRowSamplingWorkspaceSize
- Throws:
std::runtime_error – on a contract violation or a CUDA failure
- void trt_edgellm::topPProbabilitiesFromLogits(
- rt::Tensor const &logits,
- rt::Tensor &probabilities,
- float temperature,
- float topP,
- rt::Tensor &workspace,
- cudaStream_t stream
Construct the exact top-p distribution in the original vocabulary order.
- void trt_edgellm::topKLogitsToDenseProbabilities(
- rt::Tensor const &topKValues,
- rt::Tensor const &topKIndices,
- rt::Tensor &probabilities,
- int32_t vocabSize,
- float temperature,
- cudaStream_t stream
Scatter selected top-k logits into a dense probability tensor.
topKValues/topKIndices are [batch-size, top-k] outputs from selectAllTopK(). The output probabilities tensor is [batch-size, vocab-size] and is zero outside the selected top-k support.
- void trt_edgellm::selectAllTopK(
- rt::Tensor const &input,
- rt::OptionalOutputTensor topKValues,
- rt::Tensor &topKIndices,
- int32_t topK,
- rt::Tensor &workspace,
- cudaStream_t stream
Select all top-K elements from input tensor.
Returns topK indices and raw values from input with no transformations applied. This function identifies the K largest elements in each batch and returns their indices and optionally their values.
- Parameters:
input – [in] Input tensor [GPU, Float] with shape [batch-size, vocab-size]
topKValues – [out] Optional top-K values [GPU, Float] with shape [batch-size, top-K]. Can be std::nullopt if values not needed
topKIndices – [out] Top-K indices [GPU, Int32] with shape [batch-size, top-K]
topK – [in] Number of top elements to select
workspace – [inout] Workspace buffer [GPU, Int8] for intermediate computations
stream – [in] CUDA stream to execute the kernel
- Throws:
std::runtime_error – If CUDA operations fail
- void trt_edgellm::selectArgmaxAndComputeEntropy(
- rt::Tensor const &input,
- rt::Tensor &topIndices,
- rt::Tensor &entropy,
- float temperature,
- cudaStream_t stream
Select argmax and compute softmax entropy for each input row.
Returns the top-1 token ID and entropy for every row in a logits tensor. The logits are temperature-scaled for entropy computation; argmax is unchanged for positive temperatures.
- Parameters:
input – [in] Input tensor [GPU, Float/Half/BF16] with shape [rows, vocab-size]
topIndices – [out] Top-1 indices [GPU, Int32] with shape [rows, 1]
entropy – [out] Entropy values [GPU, Float] with shape [rows]
temperature – [in] Softmax temperature
stream – [in] CUDA stream to execute the kernel
- size_t trt_edgellm::getExtractTopKLogprobsWorkspaceSize(
- int32_t batchSize,
- int32_t vocabSize,
- int32_t topK
Get workspace size required for extractTopKLogprobs.
The workspace holds the intermediate log-softmax buffer ([batch-size, vocab-size] floats, 256-byte aligned) followed by the selectAllTopK temporary storage. It is NOT a plain selectAllTopK workspace; always size logprobs workspaces with this function.
- Parameters:
batchSize – [in] Batch size
vocabSize – [in] Vocabulary size
topK – [in] Number of top log-probabilities to extract
- Returns:
Required workspace size in bytes
- void trt_edgellm::extractTopKLogprobs(
- rt::Tensor const &logits,
- rt::Tensor &logprobsValues,
- rt::Tensor &logprobsIndices,
- int32_t topK,
- rt::Tensor &workspace,
- cudaStream_t stream
Extract top-K log-probabilities from raw logits.
Computes numerically-stable log-softmax over the full vocabulary for each batch element, then selects the top-K (token_id, log_prob) pairs sorted by descending probability. No temperature scaling is applied — logprobs match OpenAI convention: log(softmax(logits)).
- Parameters:
logits – [in] Input logits [GPU, Float] with shape [batch-size, vocab-size]
logprobsValues – [out] Top-K log-prob values [GPU, Float] with shape [batch-size, top-K]
logprobsIndices – [out] Top-K token indices [GPU, Int32] with shape [batch-size, top-K]
topK – [in] Number of top elements to select (must be <= kMaxLogprobsK)
workspace – [inout] Workspace buffer [GPU, Int8]; holds the log-softmax buffer and the top-K selection temp storage. Size it with getExtractTopKLogprobsWorkspaceSize().
stream – [in] CUDA stream
- Throws:
std::runtime_error – If CUDA operations fail
- void trt_edgellm::gatherSpecVerifyAcceptedLogitRows(
- rt::Tensor const &logits,
- rt::Tensor const &acceptedIndices,
- rt::Tensor &gathered,
- int32_t batchSize,
- int32_t verifyTreeSize,
- int32_t maxAcceptDepth,
- int32_t vocabSize,
- cudaStream_t stream
Spec-decode verify: gather accepted logit rows from the full verify-tree logits tensor.
For each (batch, depth) pair, copies the logit row at tree position acceptedIndices[batch * maxAcceptDepth + depth] into the output gathered tensor at row batch * maxAcceptDepth + depth. Enables running log-softmax only on the accepted rows (typically 4–7) rather than the full verify tree (60 rows). Used only by speculative-decode verification (EAGLE / MTP) when collecting per-token logprobs.
- Parameters:
logits – [in] GPU float32 [batchSize * verifyTreeSize, vocabSize]
acceptedIndices – [in] GPU int32 [batchSize * maxAcceptDepth] — tree-position indices
gathered – [out] GPU float32 [batchSize * maxAcceptDepth, vocabSize]
batchSize – [in] Active batch size
verifyTreeSize – [in] Rows per batch item in logits
maxAcceptDepth – [in] draftSteps + 1
vocabSize – [in] Vocabulary size
stream – [in] CUDA stream
- void trt_edgellm::mapReducedVocabToFullVocab( )#
Map reduced vocabulary IDs to full vocabulary IDs using a lookup table (in-place).
Performs in-place mapping from reduced vocabulary space to full vocabulary space using the provided mapping table: vocabIds[i] = vocabMappingTable[vocabIds[i]]
The operation is performed in-place, modifying the input tensor directly.
- Parameters:
vocabIds – [inout] Tensor [GPU, Int32] containing reduced vocabulary IDs as input, will be overwritten with full vocabulary IDs as output
vocabMappingTable – [in] Lookup table [GPU, Int32] with shape [reduced_vocab_size] mapping reduced IDs to full IDs
stream – [in] CUDA stream to execute the kernel
- Throws:
std::runtime_error – If CUDA operations fail
- bool trt_edgellm::operator==(
- RowSamplingParams const &lhs,
- RowSamplingParams const &rhs
- bool trt_edgellm::operator!=(
- RowSamplingParams const &lhs,
- RowSamplingParams const &rhs
- RowSamplingParams trt_edgellm::makeRowSamplingParams(
- float temperature,
- int64_t topK,
- float topP
A row’s parameters from API values.
The API sends a negative top-k to disable it; it is stored as 0 so rows that mean the same thing compare equal.
- Throws:
std::invalid_argument – if topK does not fit the kernels’ int32 top-k
Warning
doxygenfunction: Unable to resolve function “trt_edgellm::shouldUseNonGreedySampling” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:
- bool shouldUseNonGreedySampling(RowSamplingParams const ¶ms) noexcept
- bool shouldUseNonGreedySampling(float temperature, int64_t topK, float topP) noexcept