Decoder Utils#

struct ForcedPathInfo#

The forced nodes of a grafting round’s verify tree, for telling forced accepted positions from predictions.

Public Members

Tensor const *deviceRows#

[activeBatch, maxAcceptDepth] verify rows the accept kernel walked.

Tensor *hostRows#

Pinned [maxBatch, maxAcceptDepth] mirror, filled by appendAcceptedTokens.

int8_t const *hostForced#

[activeBatch, stride] 1 on a forced verify node.

int32_t stride#

Verify width of the round.

void trt_edgellm::rt::decoder_utils::zeroActiveRegion(
Tensor &tensor,
cudaStream_t stream
)#

Zero the region tensor's current shape covers, not its whole allocation.

The spec-decode hidden-state buffers are sized for max_input_len but reshaped down to the few tokens a step binds, so clearing getMemoryCapacity() would cost O(max_input_len) per token. The bound region is a contiguous prefix of the allocation, so shape.volume() is sufficient.

void trt_edgellm::rt::decoder_utils::prepareSpecRaggedBindings(
DecodingRuntimeContext &runtime,
LLMEngineConfig const &cfg,
int32_t kvCacheIndex,
Tensor const &attentionPositions,
Tensor const &committedPastLengths,
Tensor const *validCounts,
Tensor const &selectedTokenIndices,
int32_t selectedRows,
std::vector<ResidentRef> const *residentRefs,
int32_t batchSize,
int32_t queryWidth,
InferenceDims const &dims,
cudaStream_t stream
)#
RaggedExecutionBatch trt_edgellm::rt::decoder_utils::buildSpecPrefillRaggedBatch(
DecodingInferenceContext const &context,
int32_t queryWidth,
int32_t heldBackTokens
)#
ExecutionPhase trt_edgellm::rt::decoder_utils::contextPrefillPhase(
std::vector<int32_t> const &pastLengths,
int32_t activeBatchSize
)#
void trt_edgellm::rt::decoder_utils::prepareSpecPrefillRaggedBindings(
DecodingRuntimeContext &runtime,
LLMEngineConfig const &cfg,
int32_t kvCacheIndex,
DecodingInferenceContext const &context,
int32_t queryWidth,
InferenceDims const &dims,
cudaStream_t stream,
int32_t heldBackTokens
)#
Parameters:

heldBackTokens – Positions at the end of each row’s prefill this pass leaves out. A row seated into a running batch (DecodingStrategy::seatRow) holds its last prompt position back for the next decode step’s accept pass, which feeds it beside every resident’s accepted span.

void trt_edgellm::rt::decoder_utils::copyLeadingElements(
Tensor &destination,
Tensor const &source,
int64_t elements,
cudaStream_t stream
)#

Copy the leading elements elements of source into destination, of the same data type.

void trt_edgellm::rt::decoder_utils::copyHiddenRowsStrided(
Tensor &destination,
int32_t destinationStride,
Tensor const &source,
int32_t sourceStride,
int32_t batch,
int64_t hiddenSize,
cudaStream_t stream
)#

Copy batch sequences of sourceStride hidden rows each into destination at destinationStride rows per sequence (destinationStride >= sourceStride); for a decoder whose verification width varies by round and that carries the accepted spans at a fixed stride.

void trt_edgellm::rt::decoder_utils::seatAcceptedSpan(
DecodingInferenceContext const &context,
int32_t executionRow,
Tensor &acceptedTokenIds,
Tensor &acceptLength,
Tensor &hostSeatStaging,
Tensor &acceptedHiddenStates,
int32_t hiddenStride,
Tensor const &prefillHiddenStates,
cudaStream_t stream
)#

Seat an admitted row’s accepted span for a two-engine decoder whose draft trails the target by the span the last verification accepted (EAGLE, MTP).

The row’s draft prefill held its last prompt position back (heldBackTokens = 1), so between steps the row looks like a resident whose last verification accepted one token: the first token the target sampled, fed to the draft beside the target hidden state at the last prompt position. Writes that span at executionRow of the decoder’s carried tensors, staged through hostSeatStaging, a pinned [B, 2] INT32 tensor only seating writes (every seat follows the seated prefill’s synchronization, so an earlier seat’s copies have run). acceptedHiddenStates holds hiddenStride hidden rows per execution row, and prefillHiddenStates the seated prefill’s hidden states, one row per token.

void trt_edgellm::rt::decoder_utils::consumedTokenTail(
std::vector<int32_t> const &tokens,
std::vector<int32_t> const &prompt,
int32_t committedLength,
int32_t contextLength,
std::vector<int32_t> &tail
)#

The last contextLength tokens a sequence has fed to the engine, oldest first.

All of tokens but the last have been fed; the last is next round’s root. A reused prefix is missing from tokens and comes from prompt.

Parameters:
  • tokens – The prompt past any reused prefix, then the output

  • prompt – The whole prompt

  • committedLength – Committed KV length, reused prefix included

  • contextLength – Tokens to return

  • tail – [out] Shorter than contextLength only when fewer have been fed

std::unique_ptr<EngineExecutor> trt_edgellm::rt::decoder_utils::loadDraftEngine(
std::filesystem::path const &engineDir,
DeploymentConfig const &deployment,
std::shared_ptr<RtxRuntimeCache> runtimeCache = nullptr,
EngineRefitOptions refit = {}
)#

Load the draft engine from disk and return an EngineExecutor.

void trt_edgellm::rt::decoder_utils::directVocabMapToOffsets(
std::vector<int32_t> &vocabMap,
int32_t baseVocabSize
)#

Validate a DIRECT reduced-to-full vocabulary map and convert it in place to OFFSETS.

void trt_edgellm::rt::decoder_utils::appendSampledTokens(
DecodingInferenceContext &context,
int32_t const *sampledTokenIds,
int32_t activeBatchSize
)#

Append one sampled token per active slot to the host-side token lists.

Slots already marked finished are skipped: they are evicted at the end of this step, so a token appended to them is never fed back and only pollutes the output. This mirrors collectLogprobsFromHost, which skips the same slots, keeping tokenIds and stepLogprobs in step with each other. The appended token is the slot’s one pending token.

void trt_edgellm::rt::decoder_utils::appendSampledTokensForRows(
DecodingInferenceContext &context,
int32_t const *sampledTokenIds,
std::vector<int32_t> const &contextRows
)#

appendSampledTokens() over a row set: sampledTokenIds[i] is appended to context row contextRows[i]; finished rows are skipped.

float trt_edgellm::rt::decoder_utils::samplingUniformForSlot(
DecodingInferenceContext const &context,
size_t slot
) noexcept#

The uniform the slot’s next token is sampled with. A slot with requestStableSampling set draws from its seed and absolute position (full prompt + generated tokens), so the request draws the same value whichever logits row it lands on. Any other slot draws from the shared counter stream keyed on kDefaultSamplingSeed, the slot’s generated-token count and its row, as release 0.11 did for requests without a seed: such a request repeats its draws when it lands on the same row again, and rows of one batch draw different values.

void trt_edgellm::rt::decoder_utils::fillSamplingRowsForRows(
DecodingInferenceContext const &context,
std::vector<int32_t> const &contextRows,
SamplingRow *hostRows
)#

The sampling row of each selected context row, hostRows[i] for contextRows[i]: the slot’s parameters with the uniform of samplingUniformForSlot, as in prefill and vanilla decode.

bool trt_edgellm::rt::decoder_utils::enqueueSamplingForRows(
DecodingRuntimeContext &runtime,
DecodingInferenceContext &context,
Tensor &logits,
std::vector<int32_t> const &contextRows,
cudaStream_t stream
)#

Enqueue sampling of logits [contextRows.size(), vocab] for the selected context rows. Order: logit bias, the per-row sampler over each row’s own parameters and the uniform of its stream (sampleNextTokens over a row set), reduced-to-full vocabulary remap, token broadcast, logprobs staging at the widest width any selected row asks for, and the D2H of the sampled ids into runtime.sampling.hostSelectedTokenIds. Does not synchronize stream.

Returns:

False if the token broadcast failed.

void trt_edgellm::rt::decoder_utils::commitSampledRows(
DecodingRuntimeContext &runtime,
DecodingInferenceContext &context,
std::vector<int32_t> const &contextRows
)#

After the step’s synchronization, append the tokens and collect the logprobs staged by enqueueSamplingForRows() for the same contextRows.

bool trt_edgellm::rt::decoder_utils::acceptedPositionIsForced(
ForcedPathInfo const &info,
int32_t slot,
int32_t k,
int32_t maxAcceptDepth
)#

Whether accepted position k of a slot carries a forced token: the step after it continues from a forced node, whose token replaced the prediction at k. The last accepted token is always a prediction. Valid once appendAcceptedTokens has filled the host rows.

void trt_edgellm::rt::decoder_utils::appendAcceptedTokens(
DecodingInferenceContext &context,
Tensor &hostAcceptLengths,
Tensor &hostAcceptedTokenIds,
Tensor const &deviceAcceptLength,
Tensor const &deviceAcceptedTokenIds,
int32_t maxAcceptDepth,
tokenizer::Tokenizer const &tokenizer,
cudaStream_t stream,
int32_t proposedDraftsPerRound = 0,
int32_t const *perSlotProposedDrafts = nullptr,
int32_t forcedAcceptanceLength = -1,
bool proposedDraftsExact = true,
ForcedPathInfo const *forcedPath = nullptr
)#

Copy accepted tokens from device buffers into the host-side context token lists. The caller must commit the same accept lengths to persistent base state before calling this function. On return, hostAcceptLengths holds the number of tokens actually appended per slot. forcedAcceptanceLength includes the target bonus; -1 keeps natural acceptance accounting. proposedDraftsExact is false when proposedDraftsPerRound is a verify-tree capacity rather than valid proposals. forcedPath, when given, has its host rows filled under the same synchronization; the forced tokens on each slot’s path are then neither drafts nor bonus.

int32_t trt_edgellm::rt::decoder_utils::forcedAcceptanceLength(
DecodingRuntimeContext const &runtime,
DecodingInferenceContext const &context
)#

Return the forced length including the target bonus, or -1 for natural acceptance.

void trt_edgellm::rt::decoder_utils::clampAcceptLengthsToRemainingGeneration(
DecodingInferenceContext const &context,
Tensor &deviceAcceptLength,
cudaStream_t stream
)#

Clamp device accept lengths so multi-token speculative commits never exceed max_generate_length.

Runs entirely on stream: the per-slot budgets travel as kernel arguments, so the verify round has no D2H copy and no host synchronization before appendAcceptedTokens(). A mid-round host sync drains the stream, and under GPU time-slicing that forfeits a full foreign quantum every round.

void trt_edgellm::rt::decoder_utils::applyForcedAcceptance(
DecodingInferenceContext &context,
Tensor &hostAcceptLengths,
Tensor &hostAcceptedTokenIds,
Tensor &deviceAcceptLength,
Tensor &deviceAcceptedTokenIds,
std::vector<int32_t> &ownTokens,
int32_t maxAcceptDepth,
cudaStream_t stream
)#

Teacher-force this round’s acceptance to the golden’s tokens.

Call after clampAcceptLengthsToRemainingGeneration() and before the KV-cache commit, since trimming the acceptance is what keeps a replaced token’s stale cache entry out of the commit.

Parameters:

ownTokens – Out: per sequence, the token it would itself have committed at the slot that ends up last – the divergence signal the dump records.

void trt_edgellm::rt::decoder_utils::dumpSpecRound(
DecodingInferenceContext &context,
HybridCacheManager &cacheManager,
KVPageTable const &pageTable,
KVPageTable const *swaPageTable,
Tensor const &verifyLogits,
Tensor const &acceptedTokenIndices,
Tensor const &hostAcceptLengths,
std::vector<int32_t> const &ownTokens,
int32_t verifyTokenCount,
int32_t maxAcceptDepth,
cudaStream_t stream
)#

Dump one speculative round’s committed base state.

Call after appendAcceptedTokens(), so the token list and the cache are both final.

Parameters:
  • verifyLogits – Base verify logits [activeBatch, verifyTokenCount, vocab].

  • acceptedTokenIndices – Device [activeBatch, maxAcceptDepth] verify rows that were accepted.

  • hostAcceptLengths – Host accept lengths, as written back by appendAcceptedTokens().

  • ownTokens – From applyForcedAcceptance(); empty when forcing is off.

void trt_edgellm::rt::decoder_utils::enqueueLogprobsD2H(
Tensor const &inputLogits,
int32_t rows,
DecodingRuntimeContext &runtime,
int32_t topK,
cudaStream_t stream
)#

Enqueue log-softmax + top-K extraction + async D2H staging for one decode step. Device work only — results land in runtime.logprobs.host* after the caller’s round sync.

Parameters:
  • inputLogits – Row-major logits (GPU): [rows, vocabSize].

  • rows – Total rows: activeBatchSize (vanilla / prefill) or activeBatchSize * rowsPerBatch (spec decode, gathered if needed).

  • runtime – Runtime context providing logprobs/sampling buffers.

  • topK – Number of top log-probabilities to extract.

  • stream – CUDA stream (not synchronized here).

void trt_edgellm::rt::decoder_utils::collectLogprobsFromHost(
DecodingRuntimeContext &runtime,
DecodingInferenceContext &context,
int32_t activeBatchSize,
int32_t topK
)#

Collect staged logprobs into context.stepLogprobs (vanilla / prefill: one row per slot). Call after the round synchronization that followed enqueueLogprobsD2H(). The staged rows are topK wide (the batch’s widest width); each slot keeps its own width’s prefix and a slot with width zero keeps nothing.

void trt_edgellm::rt::decoder_utils::collectLogprobsFromHostForRows(
DecodingRuntimeContext &runtime,
DecodingInferenceContext &context,
std::vector<int32_t> const &contextRows,
int32_t topK
)#

collectLogprobsFromHost() over a row set: staged row i goes to context row contextRows[i]; finished rows and rows with width zero are skipped, and each row keeps its own width’s prefix. Stage with enqueueLogprobsD2H(logits, contextRows.size(), …): top-K extraction and the reduced-vocabulary index remap do not depend on which context row a logits row belongs to.

void trt_edgellm::rt::decoder_utils::collectSpecLogprobsFromHost(
DecodingRuntimeContext &runtime,
DecodingInferenceContext &context,
int32_t activeBatchSize,
int32_t rowsPerBatch,
int32_t const *hostAcceptLens,
int32_t topK
)#

Collect staged logprobs into context.stepLogprobs (multi-row decode: acceptLen rows per slot). Call after the round synchronization (appendAcceptedTokens) that made hostAcceptLens valid. Per-slot widths are handled as in collectLogprobsFromHost.

Parameters:

rowsPerBatch – Max rows per batch item: maxAcceptDepth for EAGLE/MTP, blockSize for DFlash, or canvasLen for DiffusionGemma.

void trt_edgellm::rt::decoder_utils::sampleNextTokens(
DecodingInferenceContext const &context,
int32_t activeBatchSize,
Tensor const &logits,
Tensor &selectedIndices,
Tensor &hostRows,
Tensor &deviceRows,
Tensor &workspace,
cudaStream_t stream
)#

Sample the next token of every active slot, each with its own parameters and the request-stable uniform for the position it fills, so a slot’s token never depends on its batch. Builds one SamplingRow per slot in hostRows (pinned), copies them into deviceRows and runs the per-row sampler; does not synchronize stream.

Parameters:
  • logits – [activeBatchSize, vocab] GPU float logits, one row per slot.

  • selectedIndices – GPU int32 output, reshaped here to [activeBatchSize, 1].