Vanilla Decoder#

class VanillaDecoder : public trt_edgellm::rt::DecodingStrategy#

One sampled token per step, plus grammar jump-forward.

A slot’s pending tokens (DecodingInferenceContext::pendingTokens) are the tail of its token list without KV. A plain step feeds one per slot on the decode profile. After a jump appended forced tokens, the next step is a catch-up step: every slot runs as a context-chunk row covering its pending tokens on the prefill profile, eager, and samples from its last row. Either step ends in the same sampling, grammar and logprobs tail.

Public Functions

explicit VanillaDecoder(DecodingRuntimeContext &runtime)#
inline virtual DecodingStrategyKind kind() const noexcept override#
inline virtual char const *name() const noexcept override#
inline virtual bool isSpeculative() const noexcept override#
virtual bool decodeStep(DecodingInferenceContext &context) override#
virtual bool captureCudaGraphs(cudaStream_t stream) override#
inline virtual int64_t getRequiredContextMemorySize(
) const noexcept override#
inline virtual void setContextMemory(Tensor&) override#
inline virtual void resetForNewSequences(
Tensor&,
cudaStream_t
) override#
inline virtual void onBatchEvict(
std::vector<int32_t> const&,
int32_t,
int32_t,
Tensor&,
cudaStream_t
) override#