Vanilla Decoder#
-
class VanillaDecoder : public trt_edgellm::rt::DecodingStrategy#
One sampled token per step, plus grammar jump-forward.
A slot’s pending tokens (DecodingInferenceContext::pendingTokens) are the tail of its token list without KV. A plain step feeds one per slot on the decode profile. After a jump appended forced tokens, the next step is a catch-up step: every slot runs as a context-chunk row covering its pending tokens on the prefill profile, eager, and samples from its last row. Either step ends in the same sampling, grammar and logprobs tail.
Public Functions
-
explicit VanillaDecoder(DecodingRuntimeContext &runtime)#
-
inline virtual DecodingStrategyKind kind() const noexcept override#
-
inline virtual char const *name() const noexcept override#
-
inline virtual bool isSpeculative() const noexcept override#
-
virtual bool decodeStep(DecodingInferenceContext &context) override#
-
virtual bool captureCudaGraphs(cudaStream_t stream) override#
- inline virtual int64_t getRequiredContextMemorySize(
-
explicit VanillaDecoder(DecodingRuntimeContext &runtime)#