Sampling#

struct RowSamplingParams#

Sampling parameters of one logits row, as a request carries them: the part of SamplingParams that does not depend on the launch. A plain value, so the runtime stores one per slot and rows that compare equal share a launch. topK <= 0 disables top-k.

Subclassed by trt_edgellm::SamplingParams

Public Members

float temperature = {1.0F}#
int32_t topK = {0}#
float topP = {1.0F}#
struct SamplingParams : public trt_edgellm::RowSamplingParams#

Parameters of one sampling launch: the rows’ shared RowSamplingParams plus the launch geometry.

Public Functions

inline SamplingParams(
int32_t batchSize_,
int32_t vocabSize_,
float temperature_ = 1.0f,
int32_t topK_ = 0,
float topP_ = 1.0f
)#

Constructor with default values.

With top-k disabled and top-p at 1.0 (a temperature alone) the row samples from its whole temperature-scaled distribution through the top-p path, which retains everything at p = 1.0.

Parameters:
  • batchSize_ – Number of samples in the batch

  • vocabSize_ – Size of the vocabulary

  • temperature_ – Temperature parameter (default: 1.0f)

  • topK_ – Top-K parameter (default: 0, disabled)

  • topP_ – Top-P parameter (default: 1.0f, disabled)

Throws:

std::invalid_argument – if temperature is negative

inline SamplingParams(
int32_t batchSize_,
int32_t vocabSize_,
RowSamplingParams const &row
)#

A launch over batchSize_ rows that all take row.

Public Members

int32_t batchSize#

Number of samples in the batch.

int32_t vocabSize#

Size of the vocabulary.

bool useTopK#

Flag indicating if top-K sampling is enabled.

bool useTopP#

Flag indicating if top-P sampling is enabled.

struct SamplingRow#

One row of a per-row sampling launch: a request’s parameters as the kernels read them, with the uniform the row draws.

makeSamplingRow folds a greedy row into top-k 1, so the top-k kernels serve it; topK == 0 sends a row to the top-p kernels, which sample its whole temperature-scaled distribution at topP == 1.

Public Members

float temperature = {1.0F}#
int32_t topK = {1}#
float topP = {1.0F}#
float uniform = {0.0F}#
size_t trt_edgellm::getTopKtopPSamplingWorkspaceSize(
int32_t batchSize,
int32_t vocabSize,
SamplingParams const &params
)#

Get workspace size required for top-K/top-P sampling (FP32 only).

Calculates the amount of GPU memory needed for intermediate computations during the sampling operation. The workspace must be allocated before calling topKtopPSamplingFromLogits().

Parameters:
  • batchSize – [in] Batch size for sampling

  • vocabSize – [in] Vocabulary size

  • params – [in] Sampling parameters

Throws:

std::runtime_error – if topK and topP are both not set

Returns:

Required workspace size in bytes

size_t trt_edgellm::getTopPProbabilitiesWorkspaceSize(
int32_t rows,
int32_t vocabSize
)#

Return the workspace required by topPProbabilitiesFromLogits().

size_t trt_edgellm::getSelectAllTopKWorkspaceSize(
int32_t batchSize,
int32_t vocabSize,
int32_t topK
) noexcept#

Get workspace size required for selectAllTopK operation (FP32 only).

Calculates the amount of GPU memory needed for intermediate computations during the top-K selection operation. The workspace must be allocated before calling selectAllTopK().

Parameters:
  • batchSize – [in] Batch size for selection

  • vocabSize – [in] Vocabulary size

  • topK – [in] Number of top elements to select

Returns:

Required workspace size in bytes

size_t trt_edgellm::getRowSamplingWorkspaceSize(
int32_t maxRows,
int32_t vocabSize
)#

Workspace bytes for sampleRows over at most maxRows rows: the top-k region at kMaxSamplingTopK and the top-p region, both resident because one launch may use both.

void trt_edgellm::applyLogitBias(
rt::Tensor &logits,
rt::Tensor const &tokenIds,
rt::Tensor const &biasValues,
rt::Tensor const &offsets,
cudaStream_t stream
)#

Apply sparse per-batch logit biases in place before sampling.

Biases are represented in CSR-style flattened buffers. For each batch row b, entries in [offsets[b], offsets[b + 1]) are added to logits[b, tokenIds[i]]. Invalid token IDs are ignored defensively; callers should still validate token IDs before invoking this helper.

Parameters:
  • logits – [inout] Logits tensor [GPU, Float] with shape [batch-size, vocab-size]

  • tokenIds – [in] Flattened biased token IDs [GPU, Int32] with shape [num-biased-tokens]

  • biasValues – [in] Flattened bias values [GPU, Float] with shape [num-biased-tokens]

  • offsets – [in] Per-batch CSR offsets [GPU, Int32] with shape [batch-size + 1]

  • stream – [in] CUDA stream to execute the kernel

Throws:

std::runtime_error – If tensor validation or CUDA launch fails

void trt_edgellm::applyLogitBiasRepeatedRows(
rt::Tensor &logits,
rt::Tensor const &tokenIds,
rt::Tensor const &biasValues,
rt::Tensor const &offsets,
int32_t rowsPerSlot,
cudaStream_t stream
)#

Apply sparse per-slot additive logit biases to repeated logits rows.

This variant is used by speculative verification tensors where each active request slot owns multiple contiguous logits rows. Row r consumes the CSR bias entries for slot r / rowsPerSlot.

Parameters:
  • logits – [inout] Logits tensor [GPU, Float] with shape [batch-size * rowsPerSlot, vocab-size]

  • tokenIds – [in] Flattened biased token IDs [GPU, Int32] with shape [num-biased-tokens]

  • biasValues – [in] Flattened bias values [GPU, Float] with shape [num-biased-tokens]

  • offsets – [in] Per-slot CSR offsets [GPU, Int32] with shape [batch-size + 1]

  • rowsPerSlot – [in] Number of contiguous logits rows for each active request slot

  • stream – [in] CUDA stream to execute the kernel

Throws:

std::runtime_error – If tensor validation or CUDA launch fails

void trt_edgellm::applyTokenBitmask(
rt::Tensor &logits,
rt::Tensor const &bitmask,
rt::Tensor const &rowNeedsMask,
int32_t numRows,
cudaStream_t stream
)#

Zero out grammar-forbidden tokens ahead of sampling.

The bitmask is dense and bit-packed, one bit per output-vocabulary entry, with a set bit meaning allowed. Its width equals the logits width, so no vocabulary lookup is needed. Row r of the bitmask constrains row r of the logits. Rows are not slots: each slot owns rowsPerSlot consecutive rows, exactly as in applyLogitBiasRepeatedRows.

Parameters:
  • logits – [inout] Logits [GPU, Float] with shape [num-rows, vocab-size]

  • bitmask – [in] Packed mask [GPU, Int32] with shape [>= num-rows, ceil(vocab-size / 32)]

  • rowNeedsMask – [in] Per-row flag [GPU, Int32], shape [>= num-rows]; rows set to 0 are skipped entirely, so unconstrained requests in a mixed batch cost nothing

  • numRows – [in] Number of logits rows to constrain

  • stream – [in] CUDA stream to execute the kernel

Throws:

std::runtime_error – If tensor validation or CUDA launch fails

void trt_edgellm::topKtopPSamplingFromLogits(
rt::Tensor const &logits,
rt::Tensor &selectedIndices,
SamplingParams const &params,
rt::Tensor &workspace,
cudaStream_t stream,
uint64_t randomSeed = 42,
uint64_t randomOffset = 0,
rt::Tensor const *rowUniforms = nullptr
)#

Main sampling function for top-K and top-P sampling from logits.

Performs token sampling using top-K and/or top-P (nucleus) sampling strategies on the input logits. The function applies temperature scaling and returns the selected token indices for each batch element.

Parameters:
  • logits – [in] Input logits tensor [GPU, Float] with shape [batch-size, vocab-size]

  • selectedIndices – [out] Selected token indices [GPU, Int32] with shape [batch-size, 1]

  • params – [in] Sampling parameters including batch size, vocab size, temperature, top-K, and top-P values

  • workspace – [inout] Workspace buffer [GPU, Int8] for intermediate computations

  • stream – [in] CUDA stream to execute the kernel

  • randomSeed – [in] Random seed for rows without a supplied uniform (default: 42)

  • randomOffset – [in] Random offset for rows without a supplied uniform; use a new value per decoding step (default: 0)

  • rowUniforms – [in] Optional GPU FP32 tensor [batch-size]. When supplied, each row uses this request-derived uniform instead of deriving randomness from its transient batch index.

Throws:

std::runtime_error – If CUDA operations fail

SamplingRow trt_edgellm::makeSamplingRow(
RowSamplingParams const &params,
float uniform
) noexcept#

The row the kernels run for params, drawing uniform in [0, 1).

void trt_edgellm::sampleRows(
rt::Tensor const &logits,
rt::Tensor &selectedIndices,
SamplingRow const *hostRows,
rt::Tensor const &deviceRows,
int32_t rowCount,
rt::Tensor &workspace,
cudaStream_t stream
)#

Sample one token per row, each row with its own parameters, in a fixed number of launches.

The top-k kernels run once over every row with topK > 0 and the top-p kernels once over the rest; rows that belong to the other path return at the top of each kernel. A row’s token is what the whole-batch launch of its parameters and uniform would produce (see topKtopPSamplingFromLogits), whatever else is in the batch and however the rows are ordered.

Parameters:
  • logits – [in] [GPU, Float] with shape [row-count, vocab-size]

  • selectedIndices – [out] [GPU, Int32] with shape [row-count, 1]

  • hostRows – [in] The rowCount rows, read on the host to plan the launches: every topK in [0, kMaxSamplingTopK] and every temperature non-negative

  • deviceRows – [in] [GPU, Int8] holding the same rows as rowCount SamplingRow, copied on stream

  • workspace – [inout] [GPU, Int8] sized by getRowSamplingWorkspaceSize

Throws:

std::runtime_error – on a contract violation or a CUDA failure

void trt_edgellm::topPProbabilitiesFromLogits(
rt::Tensor const &logits,
rt::Tensor &probabilities,
float temperature,
float topP,
rt::Tensor &workspace,
cudaStream_t stream
)#

Construct the exact top-p distribution in the original vocabulary order.

void trt_edgellm::topKLogitsToDenseProbabilities(
rt::Tensor const &topKValues,
rt::Tensor const &topKIndices,
rt::Tensor &probabilities,
int32_t vocabSize,
float temperature,
cudaStream_t stream
)#

Scatter selected top-k logits into a dense probability tensor.

topKValues/topKIndices are [batch-size, top-k] outputs from selectAllTopK(). The output probabilities tensor is [batch-size, vocab-size] and is zero outside the selected top-k support.

void trt_edgellm::selectAllTopK(
rt::Tensor const &input,
rt::OptionalOutputTensor topKValues,
rt::Tensor &topKIndices,
int32_t topK,
rt::Tensor &workspace,
cudaStream_t stream
)#

Select all top-K elements from input tensor.

Returns topK indices and raw values from input with no transformations applied. This function identifies the K largest elements in each batch and returns their indices and optionally their values.

Parameters:
  • input – [in] Input tensor [GPU, Float] with shape [batch-size, vocab-size]

  • topKValues – [out] Optional top-K values [GPU, Float] with shape [batch-size, top-K]. Can be std::nullopt if values not needed

  • topKIndices – [out] Top-K indices [GPU, Int32] with shape [batch-size, top-K]

  • topK – [in] Number of top elements to select

  • workspace – [inout] Workspace buffer [GPU, Int8] for intermediate computations

  • stream – [in] CUDA stream to execute the kernel

Throws:

std::runtime_error – If CUDA operations fail

void trt_edgellm::selectArgmaxAndComputeEntropy(
rt::Tensor const &input,
rt::Tensor &topIndices,
rt::Tensor &entropy,
float temperature,
cudaStream_t stream
)#

Select argmax and compute softmax entropy for each input row.

Returns the top-1 token ID and entropy for every row in a logits tensor. The logits are temperature-scaled for entropy computation; argmax is unchanged for positive temperatures.

Parameters:
  • input – [in] Input tensor [GPU, Float/Half/BF16] with shape [rows, vocab-size]

  • topIndices – [out] Top-1 indices [GPU, Int32] with shape [rows, 1]

  • entropy – [out] Entropy values [GPU, Float] with shape [rows]

  • temperature – [in] Softmax temperature

  • stream – [in] CUDA stream to execute the kernel

size_t trt_edgellm::getExtractTopKLogprobsWorkspaceSize(
int32_t batchSize,
int32_t vocabSize,
int32_t topK
) noexcept#

Get workspace size required for extractTopKLogprobs.

The workspace holds the intermediate log-softmax buffer ([batch-size, vocab-size] floats, 256-byte aligned) followed by the selectAllTopK temporary storage. It is NOT a plain selectAllTopK workspace; always size logprobs workspaces with this function.

Parameters:
  • batchSize – [in] Batch size

  • vocabSize – [in] Vocabulary size

  • topK – [in] Number of top log-probabilities to extract

Returns:

Required workspace size in bytes

void trt_edgellm::extractTopKLogprobs(
rt::Tensor const &logits,
rt::Tensor &logprobsValues,
rt::Tensor &logprobsIndices,
int32_t topK,
rt::Tensor &workspace,
cudaStream_t stream
)#

Extract top-K log-probabilities from raw logits.

Computes numerically-stable log-softmax over the full vocabulary for each batch element, then selects the top-K (token_id, log_prob) pairs sorted by descending probability. No temperature scaling is applied — logprobs match OpenAI convention: log(softmax(logits)).

Parameters:
  • logits – [in] Input logits [GPU, Float] with shape [batch-size, vocab-size]

  • logprobsValues – [out] Top-K log-prob values [GPU, Float] with shape [batch-size, top-K]

  • logprobsIndices – [out] Top-K token indices [GPU, Int32] with shape [batch-size, top-K]

  • topK – [in] Number of top elements to select (must be <= kMaxLogprobsK)

  • workspace – [inout] Workspace buffer [GPU, Int8]; holds the log-softmax buffer and the top-K selection temp storage. Size it with getExtractTopKLogprobsWorkspaceSize().

  • stream – [in] CUDA stream

Throws:

std::runtime_error – If CUDA operations fail

void trt_edgellm::gatherSpecVerifyAcceptedLogitRows(
rt::Tensor const &logits,
rt::Tensor const &acceptedIndices,
rt::Tensor &gathered,
int32_t batchSize,
int32_t verifyTreeSize,
int32_t maxAcceptDepth,
int32_t vocabSize,
cudaStream_t stream
)#

Spec-decode verify: gather accepted logit rows from the full verify-tree logits tensor.

For each (batch, depth) pair, copies the logit row at tree position acceptedIndices[batch * maxAcceptDepth + depth] into the output gathered tensor at row batch * maxAcceptDepth + depth. Enables running log-softmax only on the accepted rows (typically 4–7) rather than the full verify tree (60 rows). Used only by speculative-decode verification (EAGLE / MTP) when collecting per-token logprobs.

Parameters:
  • logits – [in] GPU float32 [batchSize * verifyTreeSize, vocabSize]

  • acceptedIndices – [in] GPU int32 [batchSize * maxAcceptDepth] — tree-position indices

  • gathered – [out] GPU float32 [batchSize * maxAcceptDepth, vocabSize]

  • batchSize – [in] Active batch size

  • verifyTreeSize – [in] Rows per batch item in logits

  • maxAcceptDepth – [in] draftSteps + 1

  • vocabSize – [in] Vocabulary size

  • stream – [in] CUDA stream

void trt_edgellm::mapReducedVocabToFullVocab(
rt::Tensor &vocabIds,
rt::Tensor const &vocabMappingTable,
cudaStream_t stream
)#

Map reduced vocabulary IDs to full vocabulary IDs using a lookup table (in-place).

Performs in-place mapping from reduced vocabulary space to full vocabulary space using the provided mapping table: vocabIds[i] = vocabMappingTable[vocabIds[i]]

The operation is performed in-place, modifying the input tensor directly.

Parameters:
  • vocabIds – [inout] Tensor [GPU, Int32] containing reduced vocabulary IDs as input, will be overwritten with full vocabulary IDs as output

  • vocabMappingTable – [in] Lookup table [GPU, Int32] with shape [reduced_vocab_size] mapping reduced IDs to full IDs

  • stream – [in] CUDA stream to execute the kernel

Throws:

std::runtime_error – If CUDA operations fail

bool trt_edgellm::operator==(
RowSamplingParams const &lhs,
RowSamplingParams const &rhs
) noexcept#
bool trt_edgellm::operator!=(
RowSamplingParams const &lhs,
RowSamplingParams const &rhs
) noexcept#
RowSamplingParams trt_edgellm::makeRowSamplingParams(
float temperature,
int64_t topK,
float topP
)#

A row’s parameters from API values.

The API sends a negative top-k to disable it; it is stored as 0 so rows that mean the same thing compare equal.

Throws:

std::invalid_argument – if topK does not fit the kernels’ int32 top-k

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::shouldUseNonGreedySampling” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:

- bool shouldUseNonGreedySampling(RowSamplingParams const &params) noexcept
- bool shouldUseNonGreedySampling(float temperature, int64_t topK, float topP) noexcept