Decoder Registry#
-
class DecoderRegistry#
Public Functions
- DecoderRegistry(
- DecodingRuntimeContext &runtime,
- DecoderRegistryInit init
- DecodingStrategy &select(
- LLMGenerationRequest const &request
-
inline DecodingStrategy &defaultDecoder() const noexcept#
The non-speculative decoder every request falls back to: what a runtime session runs, since its deployment carries no draft engine and its residents are not chosen per request.
-
bool captureCudaGraphs(cudaStream_t stream) const#
-
int64_t getRequiredContextMemorySize() const noexcept#
-
inline bool hasSpeculativeDecoder() const noexcept#
-
inline char const *speculativeDecoderName() const noexcept#
-
inline char const *defaultDecoderName() const noexcept#
Name of the non-speculative decoder (vanilla, or block_diffusion for a diffusion backbone). Exists so callers (and tests) can observe which one
select()falls back to without a speculative decoder active.
-
struct DecoderRegistryInit#
Public Members
-
std::filesystem::path engineDir#
Still needed by decoders that read strategy-specific sidecars (EAGLE’s d2t table, MTP/DFlash draft vocab maps, and DSpark head weights). Draft engine weights arrive via
draftWeightsinstead.
-
std::optional<SpecDecodeDraftingConfig> draftingConfig#
-
std::unique_ptr<EngineExecutor> draftExecutor#
Validated draft executor whose ownership is consumed by the selected speculative decoder.
-
ExternalWeightManager draftWeights#
Loaded and engine-validated draft weights, consumed by the selected speculative decoder.
-
cudaStream_t stream = {}#
-
std::filesystem::path engineDir#
- bool trt_edgellm::rt::shouldSelectDefaultDecoder(
- DecodingStrategyKind speculativeDecoderKind,
- DecodingStrategyCapabilities const &speculativeCapabilities,
- LLMGenerationRequest const &request
Return whether an available speculative decoder should yield to the default decoder for this request.