Mtp Decoder#
-
class MTPDecoder : public trt_edgellm::rt::DecodingStrategy#
Public Functions
- MTPDecoder(
- DecodingRuntimeContext &runtime,
- std::filesystem::path const &engineDir,
- std::unique_ptr<EngineExecutor> draftExecutor,
- ExternalWeightManager draftWeights,
- cudaStream_t stream
-
inline virtual DecodingStrategyKind kind() const noexcept override#
-
inline virtual char const *name() const noexcept override#
-
inline virtual bool isSpeculative() const noexcept override#
- inline virtual DecodingStrategyCapabilities capabilities(
-
virtual DecodingKvHeadroomPolicy kvHeadroom() const override#
KV headroom of the strategy’s decode rounds; the default is the vanilla one-token step.
-
virtual bool decodeStep(DecodingInferenceContext &context) override#
-
virtual bool captureCudaGraphs(cudaStream_t stream) override#
- virtual bool initializeForGeneration(
- DecodingInferenceContext &context
Hybrid+spec endpoint reuse runs the draft prefill pre-publication (mirrors EagleDecoder). Default MTP keeps its decode-round-0 draft prefill: this override is a no-op unless context.hybridSpecEndpointReuse is set.
-
virtual void beginBatch() noexcept override#
A blocking request’s batch is about to be prefilled: forget the draft progress an earlier request left behind. Unlike resetForNewSequences, KV lengths are kept: a context cache may just have restored them.
-
virtual void appendRow(int32_t executionRow) noexcept override#
A sequence joins a running batch at
executionRow(its last row). Size the per-row state now, so an eviction before seatRow() (a cancelled or failed prefill) still compacts every row.
- virtual bool seatRow(
- DecodingInferenceContext &context,
- int32_t executionRow
Seat a sequence admitted into a running batch: bring its per-row state to a resident’s between two decode steps, without touching other rows.
Called inside the batch-one projection of the sequence’s prefill, after the base model ran:
contextand every cache manager the strategy drives present the sequence as row zero, with the first generated token appended and the base outputs still in the pipeline buffers.executionRowis the sequence’s row in the full batch, where its per-row state belongs. A speculative strategy runs the draft prefill here, and must override this: the default seats only a strategy that carries no per-row state between steps.
- virtual bool prefillSeatedChunk(
- DecodingInferenceContext &context
Run the draft prefill over a non-final chunk of a sequence being seated (the stable prefix before a Hybrid+spec volatile tail), in the same projection as seatRow(). seatRow() follows after the final chunk.
- virtual std::vector<int32_t> const &commonMaterializedStateLengths(
Greatest per-slot logical prefix whose continuation state is materialized by every model in this strategy. Physical model-state tails may extend beyond this boundary. This reports decoding progress only; context-cache policy decides whether that prefix can be published.
- virtual int64_t getRequiredContextMemorySize(