LLM Engine Config#
-
struct Gemma4MTPKVSharingEntry#
Gemma4 MTP assistant-layer to target-layer shared-KV mapping.
-
struct PlePoolConfig#
Immutable external table metadata for the PLE pool/map interface.
Public Members
-
int64_t rowCount = {}#
-
int64_t rowWidth = {}#
-
double scale = {1.0}#
-
std::string checkpointDir = {}#
-
int64_t heads = {1}#
Rows per token; their widths concatenate into one embedding.
-
std::string layerMultipliersKey = {}#
int64 [ngram_size], odd position multipliers
-
std::string headVocabSizesKey = {}#
int64 [heads], pairwise-coprime moduli
-
std::string headOffsetsKey = {}#
int64 [heads], start row of each head’s slice
-
int64_t rowCount = {}#
-
struct LayerSpec#
One decoder layer’s stateful components. The table is dense over num_hidden_layers, absolutely indexed (
layers[i].absLayerIdx == i), so a bare MLP/MoE layer still occupies its slot with no kv and no states. It is the single source of truth for which states a layer carries and how each is shaped.Public Functions
-
inline bool isAttention() const noexcept#
-
bool isRecurrent() const noexcept#
Carries mixer state, i.e. a state other than
kNgramConv.
Public Members
-
int32_t absLayerIdx = {}#
-
std::optional<KVLayerConfig> kv = {}#
Set when the layer attends; absent for a purely recurrent layer. “Is attention” := has kv.
-
int32_t kvSharingDonor = {-1}#
LOCAL attention index donating this layer’s KV, or -1 when it owns its own.
-
std::vector<StateSpec> states = {}#
Per-slot states this layer carries, at most one per kind. Each
shapeis one slot’s row: the pool prepends its own slot count (seeRecurrentStateManager).
-
inline bool isAttention() const noexcept#
-
struct LLMEngineConfig#
Parsed model and engine capabilities shared across runtime components.
Public Functions
-
inline int32_t ngramHeads() const noexcept#
Row IDs looked up per token: one per head, over every order but unigram.
-
inline int32_t ngramContextLength() const noexcept#
Tokens of history the hash reaches back over, and therefore the state carried across calls.
-
inline int32_t ngramConvStateLen() const noexcept#
Steps of convolution state PLE carries per sequence.
-
inline bool ngramEnabled() const noexcept#
-
inline bool usesPlePool() const noexcept#
Whether this engine binds the shared PLE row pool; add new pool users here, not at call sites.
-
bool supportsBoundedSwaKVCache() const#
Whether exported layer metadata declares bounded SWA storage capability.
-
bool usesBoundedSwaKVCache() const#
Whether this runtime instance actively uses bounded SWA storage.
-
void setSwaKVCacheMode(SwaKVCacheMode mode) noexcept#
Select the runtime storage policy without changing exported capability metadata.
-
int32_t getSwaKVCacheModeInputLength() const#
Shape-only mode input length: 1 for bounded storage and 0 for full storage.
- int32_t getBoundedKVPoolPagesForLayer(
- KVLayerConfig const &layerConfig
Bounded-capable profile page count for one layer, independent of the active runtime policy.
- std::array<int32_t, 3> getKVPoolPageProfileForLayer(
- KVLayerConfig const &layerConfig
Static engine profile selectors: min, smaller-policy opt, and max across bounded and full page counts.
- int32_t getKVPoolPagesForLayer(
- KVLayerConfig const &layerConfig
Active physical page count for one layer’s KV pool binding.
-
std::vector<int32_t> flattenedSkipSoftmaxPrefillScaleTable() const#
skipSoftmaxPrefillScaleTable flattened to the INT32 [bin_0, S_0, bin_1, S_1, …] binding payload.
-
std::vector<int32_t> flattenedSkipSoftmaxDecodeScaleTable() const#
skipSoftmaxDecodeScaleTable flattened the same way.
-
int64_t skipSoftmaxPrefillScaleTableLen() const#
Element count of the full
skip_softmax_prefill_scalebinding (0 = dense prefill).
-
int64_t skipSoftmaxDecodeScaleTableLen() const#
Element count of the
skip_softmax_decode_scalebinding (0 = dense decode).
- int64_t skipSoftmaxPrefillScaleLen(
- int64_t seqLen,
- ExecutionPhase phase
skip_softmax_prefill_scale binding length for a prefill of
seqLennew tokens: 0 (dense cubin, zero tax) when the whole batch is known to sit below the table’s first skippable length, else the full table. Chunked prefill (kContextChunk) has KV history, so its total length is unknown here: the table is bound and the kernel’s per-sequence guard decides.
- InferenceDims prefillDims(
- int64_t batch,
- int64_t seqLen,
- ExecutionPhase phase
Prefill dims (vanilla LLM, SpecDecode base, and SpecDecode draft). seqLen is the prompt length being processed this step. kvCacheAllEmpty signals whether this is the initial prefill of an empty KV cache — this drives autoregressive engines’
kvcache_start_indexshape to[0](engine’s “initial prefill” sentinel) instead of[batch]. DiffusionGemma keepskvcache_start_indexat[batch]and usescontext_mask_selectoras its attention-mask sentinel.
-
InferenceDims decodeDims(int64_t batch) const#
Vanilla single-token decode dims. seqLen is always 1 here; packedMaskLen is 1 (no proposal mask in vanilla).
- InferenceDims compactDims(
- int64_t numSequences,
- int64_t numTokens,
- int64_t numContextSequences,
- int64_t numLogits
Compact phase-8 dims: numSequences rows packed into numTokens tokens, context sequences first. numContextSequences is the carrier extent Np (0 only for a decode-only subset, where numTokens must equal numSequences); numLogits is the logits row count L in [1, numSequences], placeholder rows included (the compact builder emits L = numSequences). Execute on profile compactStepProfileIndex(Np) (engineExecutor.h).
-
InferenceDims denoiseDims(int64_t batch, int64_t canvasLen) const#
DiffusionGemma denoise dims. Uses existing chunked-prefill execution: seqLen/selectLen are the canvas length, kvcache_start_index is [batch], and context_mask_selector is [batch] to select non-causal PADDING mask.
- InferenceDims diffusionCommitDims(
- int64_t batch,
- int64_t commitLen
DiffusionGemma commit dims. Reruns finalized tokens causally before KV commit; context_mask_selector is [0], so AttentionPlugin keeps causal mask.
- InferenceDims specVerifyDims(
- int64_t batch,
- int64_t verifyTokenCount,
- SpecVerifyLayout verifyLayout
SpecDecode base verification dims. verifyTokenCount feeds three fields: seqLen, selectLen, and packedMaskLen. This is the only recipe where selectLen != 1.
- InferenceDims proposalDims(
- int64_t batch,
- int64_t proposalSize,
- int64_t draftTopK
SpecDecode draft proposal dims. proposalSize feeds seqLen, attnMaskSeqLen, and packedMaskLen. draftTopK feeds selectLen — the draft proposal selects draftTopK tokens per sequence, matching the 3D logits output shape [batch, draftTopK, draftVocabSize].
-
InferenceDims acceptDims(int64_t batch, int64_t acceptLen) const#
SpecDecode draft accept-token dims. acceptLen is in [1, maxPathLength]; feeds seqLen and packedMaskLen.
-
InferenceDims resetDims() const#
Clean-slate dims for CUDA-graph capture reset. All 1s except kvLen = maxKVCacheCapacity. ropeBatch is fixed at 1 even for MRope models — this matches the pre-migration behavior in both runtimes (reset is a binding placeholder, not an inference step).
Public Members
Model hidden dimension.
-
int32_t outputVocabSize = {}#
Actual output vocab (reduced if vocab reduction active)
-
int32_t numAttentionLayers = {}#
Number of attention layers needing KV cache.
-
int32_t numKVHeads = {}#
Number of key-value heads.
-
int32_t numAttentionHeads = {}#
Number of query heads; 0 when the config omits num_attention_heads
-
int32_t indexerHeadDim = {}#
Width of the Qwen Sparse Attention indexer state on each KV pool row; > 0 when every attention layer is a QsaAttentionPlugin layer, 0 when the config omits indexer_head_dim.
-
int32_t headDim = {}#
Dimension of each attention head.
-
int32_t maxSupportedBatchSize = {}#
Maximum supported batch size.
-
int32_t tpSize = {1}#
Tensor-parallel size the engine was built for.
-
int32_t maxSupportedInputLength = {}#
Maximum supported input length.
-
int32_t maxKVCacheCapacity = {}#
Maximum KV cache capacity (sequence length)
-
RaggedBackendKind raggedBackend = {RaggedBackendKind::kNone}#
-
int32_t maxNumSequences = {}#
-
int32_t maxQueryLength = {}#
-
int32_t maxPhysicalTokens = {}#
-
int32_t recurrentPoolRows = {}#
-
bool compactExecution = {false}#
builder_config.compact_execution is true: every compact-capable plugin layer reserved the compact workspace
-
std::vector<std::array<int64_t, 2>> skipSoftmaxPrefillScaleTable#
Prefill skip-softmax scale table (
skip_softmax_prefill_scale_table): rows of (seqlen_bin, S) sorted by bin, empty = dense prefill. Uploaded once to theskip_softmax_prefill_scalebinding; the FMHA kernel floor-looks-up S by each sequence’s KV length. A flat S is the one-row table {0, S}.
-
std::vector<std::array<int64_t, 2>> skipSoftmaxDecodeScaleTable#
Decode skip-softmax scale table (
skip_softmax_decode_scale_table), same row format. Uploaded once to theskip_softmax_decode_scalebinding; empty = dense decode.
-
int32_t kvPoolPages = {}#
Exact physical K-page count serialized in KV binding shapes
-
int32_t numSwaPages = {}#
Total physical K-page budget for the independent SWA pool. Zero means no reduced pool; reduced markers require a positive persisted value with bounded publication headroom.
-
SwaKVCacheMode swaKVCacheMode = {SwaKVCacheMode::kBounded}#
Runtime-only storage selection. The exported per-layer capacity markers remain unchanged.
-
int32_t rotaryDim = {}#
Rotary embedding dimension.
-
int32_t numDecoderLayers = {}#
Total decoder layers (attention + linear)
-
int32_t vocabSize = {}#
Full vocabulary size.
-
int32_t reducedVocabSize = {0}#
0 = no vocab reduction
-
int32_t diffusionCanvasLength = {0}#
-
int32_t diffusionMaxDenoisingSteps = {0}#
-
int32_t diffusionSelfConditioningSize = {0}#
-
float diffusionTMax = {0.8F}#
-
float diffusionTMin = {0.4F}#
-
float diffusionEntropyBound = {0.1F}#
-
float diffusionEntropyThreshold = {0.005F}#
-
float rmsNormEps = {1.0e-6F}#
-
int32_t diffusionStabilityWindow = {2}#
-
bool isSpecDecodeBase = {false}#
Base engine exposes speculative decoding verification bindings.
-
bool isDiffusionBackbone = {false}#
DiffusionGemma phase-aware transformer backbone engine.
-
bool diffusionUnifiedConditioning = {false}#
Backbone engine owns DiffusionGemma self-conditioning inputs.
-
bool contextMaskSelectorEnabled = {false}#
Engine exposes context_mask_selector binding.
-
SpecDecodeMode specDecodeType{SpecDecodeMode::kNONE}#
Speculative decoding strategy mode (parsed from spec_decode_type)
-
trt_edgellm::DataType kvCacheDtype = {trt_edgellm::DataType::kHALF}#
KV cache data type. Parsed from required top-level
kv_cache_dtypeinconfig.json(written byllm_export.py). Accepted values: “fp16” → kHALF, “fp8” → kFP8, “int8” → kINT8, “bf16” → kBF16. The runtime validates this against the engine’s actual KV binding dtype.
-
std::vector<std::array<float, 2>> kvCacheScales#
Per-attention-layer K/V scales that convert FP8 KV values back to FP16. Empty when the export does not provide scales for an external KV consumer.
-
trt_edgellm::DataType recurrentStateDtype = {trt_edgellm::DataType::kHALF}#
Recurrent state data type (hybrid models only). Parsed from required top-level
recurrent_state_dtypewhennumLinearAttnLayers > 0; left at the default otherwise. Runtime validates against the engine’s recurrent-state binding dtype.
-
trt_edgellm::DataType convStateDtype = {trt_edgellm::DataType::kHALF}#
Conv state data type (hybrid models only). Same shape as
recurrentStateDtype.
-
RopeConfig ropeConfig = {}#
Full RoPE configuration.
-
bool useDualRope = {false}#
Use separate RoPE caches for sliding/full attention.
-
RopeConfig slidingRopeConfig = {}#
RoPE configuration for sliding attention layers.
-
RopeConfig fullRopeConfig = {}#
RoPE configuration for full attention layers.
-
int32_t slidingRotaryDim = {}#
Rotary dimension for sliding attention RoPE.
-
int32_t fullRotaryDim = {}#
Rotary dimension for full attention RoPE.
-
bool useContextDependentRope = {false}#
Use context-dependent RoPE.
-
int32_t numDeepstackFeatures = {0}#
Deepstack features (Qwen3-VL/Qwen3-Omni)
-
int32_t maxSupportedLoraRank = {0}#
Maximum LoRA rank (0 = no LoRA)
-
PlePoolConfig plePool = {}#
External immutable table for the pool/map ABI.
-
bool gemmaPleEnabled = {false}#
One embedding slice per decoder layer.
-
int32_t gemmaPleLayerCount = {0}#
Number of slices (0 = disabled)
-
int32_t gemmaPleHiddenSize = {0}#
Hidden dimension of each slice (0 = disabled)
-
std::vector<int32_t> ngramLayers = {}#
Injected decoder layers, zero-indexed; empty = disabled.
-
int32_t ngramEmbedDim = {0}#
Width of the gathered n-gram vector handed to the engine.
-
int32_t ngramConvKernel = {0}#
Kernel of the dilated depthwise conv; dilation is ngramSize.
-
int32_t ngramConvDim = {0}#
Channels of that conv; spans every residual stream.
-
int32_t ngramSize = {0}#
Longest n-gram order hashed into a row ID.
-
int32_t ngramHeadsPerOrder = {0}#
Hash heads per n-gram order
-
trt_edgellm::DataType ngramConvStateDtype = {trt_edgellm::DataType::kHALF}#
Data type of the injection’s dilated-conv state. Independent of
convStateDtype: the injected layer need not be recurrent, so a model may bind this state and no other.
-
std::vector<int32_t> eosTokenIds = {}#
Additional EOS token IDs parsed from config.json
eos_token_idarray. Models like Gemma4 have multiple EOS tokens (e.g. [1, 106] for <eos> and <turn|>). Empty ifeos_token_idis absent or scalar.
-
int32_t padTokenId = {-1}#
PLE lookup token ID for multimodal positions (-1 = unused)
-
int32_t imageTokenId = {-1}#
Special token ID for image (-1 = unused)
-
int32_t audioTokenId = {-1}#
Special token ID for audio (-1 = unused)
-
bool useVisionBidirectionalAttention = {false}#
Gemma4 Unified prefill uses block-causal image attention.
-
int32_t numLinearAttnLayers = {0}#
Number of linear attention / recurrent layers.
-
int32_t recurrentStateNumHeads = {0}#
Recurrent state heads (hv for GDN, mamba_num_heads for Mamba)
-
int32_t recurrentStateHeadDim = {0}#
Recurrent state head dimension.
-
int32_t recurrentStateSize = {0}#
Recurrent state dimension (v for GDN, dstate for Mamba)
-
int32_t recurrentStateNumGroups = {0}#
Mamba B/C group count (spec-verify replay-B extent; 0 for GDN)
-
bool recurrentSpecVerifyUsesReplay = {false}#
Selects Mamba factorized replay storage, parsed from
recurrent_spec_verify_mode(“replay” → true, “snapshot”/absent → false).
-
int32_t convDim = {0}#
Conv1d channel dimension.
-
int32_t convKernel = {0}#
Conv1d kernel width.
-
int32_t maxSpecVerifyTokens = {0}#
Max seq_len the base engine accepts for proposal verification. Parsed from
builder_config.max_spec_verify_tokenswhenisSpecDecodeBase == true; 0 otherwise. Consumers prefer the consolidatedDeploymentConfig::specDecodewhen the deployment view is available.
-
int32_t maxSpecDraftTokens = {0}#
Max seq_len the draft engine accepts for proposal / draft generation. Parsed from
builder_config.max_spec_draft_tokensbyparseDraftEngineConfig; 0 on base / vanilla engines. Consumers prefer the consolidatedDeploymentConfig::specDecodewhen the deployment view is available.
-
int32_t baseModelHiddenSize = {0}#
Hidden dim the draft engine expects for its
hidden_states_inputbinding (== the base engine’s hidden-state output dim as seen by the draft). Parsed from top-levelbase_model_hidden_sizebyparseDraftEngineConfig; left at 0 on base / vanilla engines. Differs frombase.hiddenSizefor EAGLE-3 (= base.hiddenSize * 3, multi-layer concat) and equalsbase.hiddenSizefor MTP. The deployment factory copies this intoDeploymentConfig::specDecode->baseOutputHiddenDim.
-
int32_t specDraftBlockSize = {0}#
Cached draft proposal block size for DFlash/JetSpec/DSpark. Parsed from the mode-specific config object (
dflash_config,jetspec_config, ordspark_config).
-
int32_t specDraftMaskTokenId = {0}#
Mask token ID used to seed cached draft input blocks for DFlash/JetSpec/DSpark.
-
DFlashVersion dflashVersion = {DFlashVersion::kV1}#
Versioned DFlash engine/runtime ABI. Missing metadata defaults to V1.
-
DFlashProjector dflashProjector = {DFlashProjector::kSelector}#
-
int32_t specLiLiCorrTopK = {0}#
LiLiCorr candidate count per proposal slot (
dflash_config.lilicorr_candidate_topk); 0 for the selector.
-
int32_t specSelectorTopK = {0}#
-
int32_t specSelectorRank = {0}#
-
std::string dflash2SelectorFile#
-
int32_t specConvKernelSize = {0}#
-
int32_t specConvGroupSize = {0}#
-
bool specSupportsProbabilistic = {false}#
-
int32_t dominoGruHiddenDim = {0}#
Domino runtime correction head metadata (cached block draft + GRU rollout).
-
int32_t dominoProjDim = {0}#
-
int32_t dominoPrefixLen = {0}#
-
int32_t dominoPoolSize = {0}#
-
std::string dominoHeadsFile = {}#
-
int32_t draftCandidateCount = {0}#
EAGLE draft candidate head (
eagle3_config.candidate_head): candidates per row emitted asspec_proposal_support_*instead oflogits(0 = dense logits head).
-
bool specDraftCausalHead = {false}#
Whether cached draft proposal self-attention is causal. JetSpec uses causal rows.
-
std::vector<int32_t> specTargetLayerIds = {}#
Target decoder-layer IDs whose hidden states are concatenated for cached drafts. EAGLE base engines own this contract through
eagle_hidden_state_layers; DFlash/DSpark keep their mode-specific target-layer metadata on the engine that exports it.
-
std::string modelType#
Top-level model type string, if exported.
Draft assistant reads target KV cache.
-
bool hasOwnKVCache = {true}#
Draft assistant owns/writes its own KV cache.
-
bool constantDraftPositions = {false}#
Assistant draft position IDs stay constant through a chain.
-
bool returnsFeedbackHidden = {false}#
Assistant emits backbone-space feedback hidden states.
-
int32_t assistantHiddenSize = {0}#
Assistant internal hidden dim, when distinct from hiddenSize.
-
std::vector<Gemma4MTPKVSharingEntry> gemma4MTPKVSharingMap = {}#
Assistant -> target KV sharing map.
-
bool dsparkEnableConfidenceHead = {false}#
DSpark Markov/confidence sidecar metadata.
-
bool dsparkConfidenceHeadWithMarkov = {false}#
-
std::string dsparkMarkovHeadType = {}#
-
int32_t dsparkMarkovRank = {0}#
-
std::string dsparkHeadsFile = {}#
-
std::string dsparkHeadsInfoFile = {}#
-
bool dsparkContiguousQuerySwa = {false}#
Whether the draft engine baked the contiguous-query sliding-window XQA variant, which reconstructs each query row’s position as firstQueryPosition + queryRow. This describes the draft engine query ABI; it does not gate the base engine’s DDTree verification topology.
-
bool dsparkSampleFromAnchor = {true}#
When true the anchor slot itself is a proposal and the draft query block is
block_sizewide; when false slot 0 is the bonus token, the block carries one extra mask slot and proposals start at slot 1.
-
std::vector<HybridCacheManager::LayerType> layerTypes = {}#
Absolute decoder-layer -> attention|mamba. Populated either from the canonical
layer_typesfield in config.json or by broadcasting scalarnumAttentionLayers/numLinearAttnLayersfor back-compat. Size equals the number of stateful decoder layers (mlp/moe excluded on the Python side).
-
std::vector<LayerSpec> layers = {}#
Per-layer stateful components, dense over num_hidden_layers, absolutely indexed (
layers[i].absLayerIdx == i). Restates thelayerTypes/kvLayerConfigs/kvSharingDonorsarrays and the global recurrent/n-gram geometry in one table, so a consumer can iterate a layer’s states without knowing which mixer produced them. Empty when derivation cannot recover absolute indices (e.g. a compact-indexedlayer_typesthat drops mlp/moe layers).
-
std::vector<KVLayerConfig> kvLayerConfigs = {}#
Per-attention-layer KV config. Size equals the attention count in
layerTypes. Indexed by LOCAL attention-layer index (0..numAttn-1), NOT absolute decoder-layer index.
-
std::vector<int32_t> kvSharingDonors = {}#
Per-attention-layer KV sharing donor index. Size equals the attention count in
layerTypes. Value of -1 means the layer owns its own KV cache (normal). A value >= 0 means this layer shares the KV cache from the donor attention layer at that LOCAL index (the donor’s cache is bound to this layer’s KV input). Used for Gemma4’s KV sharing where the last N layers reuse a donor’s cache.
-
inline int32_t ngramHeads() const noexcept#
- char const *trt_edgellm::rt::specDecodeModeName(
- SpecDecodeMode mode
- bool trt_edgellm::rt::isCachedBlockDraftMode(
- SpecDecodeMode mode
- bool trt_edgellm::rt::isCachedBlockDraftBase(
- LLMEngineConfig const &config
- bool trt_edgellm::rt::isCachedBlockDraftDraft(
- LLMEngineConfig const &config
- LLMEngineConfig trt_edgellm::rt::parseEngineConfig(
- std::filesystem::path const &configPath,
- std::optional<int32_t> rank,
- std::optional<int32_t> expectedWorldSize
Parse a
config.jsonfile (the same format used by the existing runtime) into anLLMEngineConfig.- Parameters:
configPath – Path to
config.json.- Throws:
std::runtime_error – if file cannot be opened/parsed or required fields are missing.
- Returns:
Parsed configuration.
- LLMEngineConfig trt_edgellm::rt::parseDraftEngineConfig(
- std::filesystem::path const &configPath
Parse a SpecDecode draft engine’s
config.jsoninto anLLMEngineConfig.The draft config carries a reduced field set (no
builder_config.spec_base, its owndraft_vocab_size).max_spec_draft_tokensis required and is parsed intocfg.maxSpecDraftTokens;cfg.maxSpecVerifyTokensstays at 0 on the draft side.isSpecDecodeBaseis left false because this is the draft — not the base — engine. Cross-engine fields (draftHiddenSize,baseOutputHiddenDim) are not stored onLLMEngineConfig; they are derived increateDeploymentConfigand live onDeploymentConfig::specConfig.- Parameters:
configPath – Path to the draft engine’s
config.json.- Throws:
std::runtime_error – on parse failure or missing required fields.
- Returns:
Parsed configuration.
- std::string trt_edgellm::rt::formatEngineConfig(
- LLMEngineConfig const &config
Format the config as a human-readable string (for logging).
- void trt_edgellm::rt::validateAgainstEngine(
- LLMEngineConfig const &config,
- EngineExecutor const &executor,
- char const *engineLabel
Cross-check an engine’s KV / recurrent / conv binding dtypes against their parsed-config counterparts. The parsed config is the source of truth; this raises a runtime_error with an actionable message on any mismatch.
- Parameters:
config – Parsed
LLMEngineConfig(fromparseEngineConfigor similar).executor – Engine whose binding dtypes should match.
engineLabel – Short label (“base” / “draft”) used in error messages.
- Throws:
std::runtime_error – on any KV / recurrent / conv dtype mismatch.