Spec Decode Utils#

struct SpecVerifyInputs#

Borrowed root-inclusive candidates and metadata for one target verification.

Public Members

Tensor const &tokenIds#

Tensor const *executionValidTokenCounts#

Tensor const *parentIds#

Tensor const *depths#

int32_t requestCount#

int32_t tokensPerRequest#

int32_t maxPathLength#

SpecVerifyLayout layout#

bool metadataReady = {false}#

Tensor const *forcedFlags = {nullptr}#: INT8 flags at the verify stride, 1 on a grafted forced token; null when the round grafted none.

struct SpecAcceptResult#

Borrowed accepted tokens; row indices address the verification stride.

Public Members

Tensor const &tokenIds#

Tensor const &emittedTokenCounts#

Tensor const *acceptedRowIndices#

int32_t outputStride#

struct GreedyAcceptInputs#

Public Members

SpecVerifyInputs verify#

Tensor const &targetTokenIds#

Tensor const *acceptValidTokenCounts#

Tensor const *forcedFlags = {nullptr}#: Per-candidate INT8 flags at the verify stride; a flagged child is accepted in place of the prediction.

struct DenseProbabilityAcceptInputs#

Public Members

Tensor const &targetProbabilities#

Tensor const &draftProbabilities#

Tensor const &proposalTokenIds#

Tensor const &proposalLengths#

Tensor const &uniforms#

int32_t batchSize#

int32_t draftStride#

int32_t verifyProposalLen#

int32_t vocabSize#

struct SparseProbabilityAcceptInputs#

Public Members

Tensor const &targetProbabilities#

Tensor const &targetSupportIds#

Tensor const &draftProbabilities#

Tensor const &draftSupportIds#

Tensor const &proposalTokenIds#

Tensor const &proposalLengths#

Tensor const &uniforms#

Tensor const *maxAcceptLengths#

struct DenseTargetSparseDraftAcceptInputs#

Public Members

Tensor const &targetProbabilities#

Tensor const &draftProbabilities#

Tensor const &draftSupportIds#

Tensor const &proposalTokenIds#

Tensor const &proposalLengths#

Tensor const &uniforms#

Tensor const *maxAcceptLengths#

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::rt::spec_decode::accept” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:

- void accept(DenseProbabilityAcceptInputs const &inputs, Tensor &ids, Tensor &counts, cudaStream_t stream, int32_t forcedAcceptanceLength)
- void accept(DenseTargetSparseDraftAcceptInputs const &inputs, Tensor &ids, Tensor &counts, Tensor *rows, cudaStream_t stream, int32_t forcedAcceptanceLength)
- void accept(GreedyAcceptInputs const &inputs, Tensor &ids, Tensor &counts, Tensor *rows, cudaStream_t stream, int32_t forcedAcceptanceLength)
- void accept(SparseProbabilityAcceptInputs const &inputs, Tensor &ids, Tensor &counts, Tensor *rows, cudaStream_t stream, int32_t forcedAcceptanceLength)

void trt_edgellm::rt::spec_decode::commitAcceptedTargetPath( DecodingRuntimeContext &runtime, DecodingInferenceContext const &context, SpecAcceptResult const &accepted, int32_t verifyTokenCount, std::optional<SpecVerifyLayout> layoutOverride )#

Commit a clamped prefix or indexed path according to the resolved verification layout. Hidden rows retain verifyTokenCount stride for chains and are packed to outputStride for trees.

Parameters:: layout – The layout the path was accepted in; the deployment’s when absent. A round that grafted forced tokens is indexed even on a chain deployment.

void trt_edgellm::rt::spec_decode::validateSpecVerifyInputs( SpecVerifyInputs const &inputs, SpecDecodeConfig const &config )#: Validate host-visible shape, device, dtype and configured capacity before enqueue.

void trt_edgellm::rt::spec_decode::prepareTargetVerifyInputs( SpecVerifyInputs const &inputs, DecodingRuntimeContext &runtime, DecodingInferenceContext const &context, int32_t targetKvCacheIndex )#: Prepare existing target bindings without executing the target or committing state.