Gemma4 Mtp Decoder#
-
class Gemma4MTPDecoder : public trt_edgellm::rt::DecodingStrategy#
Gemma4 assistant MTP decoder.
This class owns the Gemma4-specific draft-engine binding path. The assistant has no draft KV cache: its
past_key_values_*inputs are zero-copy aliases to the base target KV cache selected bykv_sharing_map.Public Functions
- Gemma4MTPDecoder(
- DecodingRuntimeContext &runtime,
- std::unique_ptr<EngineExecutor> draftExecutor,
- ExternalWeightManager draftWeights,
- cudaStream_t stream
-
inline virtual DecodingStrategyKind kind() const noexcept override#
-
inline virtual char const *name() const noexcept override#
-
inline virtual bool isSpeculative() const noexcept override#
- inline virtual DecodingStrategyCapabilities capabilities(
-
virtual DecodingKvHeadroomPolicy kvHeadroom() const override#
KV headroom of the strategy’s decode rounds; the default is the vanilla one-token step.
-
virtual bool decodeStep(DecodingInferenceContext &context) override#
-
virtual bool captureCudaGraphs(cudaStream_t stream) override#
-
virtual void beginBatch() noexcept override#
A blocking request’s batch is about to be prefilled: forget the draft progress an earlier request left behind. Unlike resetForNewSequences, KV lengths are kept: a context cache may just have restored them.
-
virtual void appendRow(int32_t executionRow) noexcept override#
A sequence joins a running batch at
executionRow(its last row). Size the per-row state now, so an eviction before seatRow() (a cancelled or failed prefill) still compacts every row.
- virtual bool seatRow(
- DecodingInferenceContext &context,
- int32_t executionRow
Seat a sequence admitted into a running batch: bring its per-row state to a resident’s between two decode steps, without touching other rows.
Called inside the batch-one projection of the sequence’s prefill, after the base model ran:
contextand every cache manager the strategy drives present the sequence as row zero, with the first generated token appended and the base outputs still in the pipeline buffers.executionRowis the sequence’s row in the full batch, where its per-row state belongs. A speculative strategy runs the draft prefill here, and must override this: the default seats only a strategy that carries no per-row state between steps.
- virtual int64_t getRequiredContextMemorySize(