LLM Engine Config

struct LLMEngineConfig

Unified configuration for base, vanilla decode, and SpecDecode draft engines.

Replaces LLMEngineRunnerConfig and EagleDraftEngineRunnerConfig with a single structure that can be parsed once and used across all runtime components (KV cache, RoPE, LoRA, preprocessors, etc.).

Public Functions

InferenceDims prefillDims(

int64_t batch,

int64_t seqLen,

bool kvCacheAllEmpty

) const

Prefill dims (vanilla LLM, SpecDecode base, and SpecDecode draft). seqLen is the prompt length being processed this step. kvCacheAllEmpty signals whether this is the initial prefill of an empty KV cache — for plugin-path engines, this drives the kvcache_start_index shape to [0] (engine’s “initial prefill” sentinel) instead of [batch]. TRT-native-ops engines always use [batch] regardless.

InferenceDims decodeDims(int64_t batch) const

Vanilla single-token decode dims. seqLen is always 1 here; packedMaskLen is 1 (no proposal mask in vanilla).

InferenceDims specVerifyDims(int64_t batch, int64_t verifySize) const

SpecDecode base verification dims. verifySize feeds three fields: seqLen, selectLen, and packedMaskLen. This is the only recipe where selectLen != 1.

InferenceDims proposalDims(

int64_t batch,

int64_t proposalSize,

int64_t draftTopK

) const

SpecDecode draft proposal dims. proposalSize feeds seqLen, attnMaskSeqLen, and packedMaskLen. draftTopK feeds selectLen — the draft proposal selects draftTopK tokens per sequence, matching the 3D logits output shape [batch, draftTopK, draftVocabSize].

InferenceDims acceptDims(int64_t batch, int64_t acceptLen) const

SpecDecode draft accept-token dims. acceptLen is in [1, draftingStep+1]; feeds seqLen and packedMaskLen.

InferenceDims resetDims() const

Clean-slate dims for CUDA-graph capture reset. All 1s except kvLen = maxKVCacheCapacity. ropeBatch is fixed at 1 even for MRope models — this matches the pre-migration behavior in both runtimes (reset is a binding placeholder, not an inference step).

Public Members

int32_t hiddenSize = {}

Model hidden dimension.

int32_t outputVocabSize = {}

Actual output vocab (reduced if vocab reduction active)

int32_t numAttentionLayers = {}

Number of attention layers needing KV cache.

int32_t numKVHeads = {}

Number of key-value heads.

int32_t headDim = {}

Dimension of each attention head.

int32_t maxSupportedBatchSize = {}

Maximum supported batch size.

int32_t maxSupportedInputLength = {}

Maximum supported input length.

int32_t maxKVCacheCapacity = {}

Maximum KV cache capacity (sequence length)

int32_t rotaryDim = {}

Rotary embedding dimension.

int32_t numDecoderLayers = {}

Total decoder layers (attention + linear)

int32_t vocabSize = {}

Full vocabulary size.

int32_t reducedVocabSize = {0}

0 = no vocab reduction

bool useTrtNativeOps = {false}

Use TRT native ops instead of custom plugin.

bool isSpecDecodeBase = {false}

Base engine exposes speculative decoding verification bindings.

SpecDecodeMode specDecodeType{SpecDecodeMode::kNONE}

Speculative decoding strategy mode (parsed from spec_decode_type)

nvinfer1::DataType kvCacheDtype = {nvinfer1::DataType::kHALF}

KV cache data type. Parsed from required top-level kv_cache_dtype in config.json (written by llm_export.py). Accepted values: “fp16” → kHALF, “fp8” → kFP8, “int8” → kINT8, “bf16” → kBF16. The runtime validates this against the engine’s actual KV binding dtype.

nvinfer1::DataType recurrentStateDtype = {nvinfer1::DataType::kHALF}

Recurrent state data type (hybrid models only). Parsed from required top-level recurrent_state_dtype when numLinearAttnLayers > 0; left at the default otherwise. Runtime validates against the engine’s recurrent-state binding dtype.

nvinfer1::DataType convStateDtype = {nvinfer1::DataType::kHALF}

Conv state data type (hybrid models only). Same shape as recurrentStateDtype.

RopeConfig ropeConfig = {}

Full RoPE configuration.

bool useDualRope = {false}

Use separate RoPE caches for sliding/full attention.

RopeConfig slidingRopeConfig = {}

RoPE configuration for sliding attention layers.

RopeConfig fullRopeConfig = {}

RoPE configuration for full attention layers.

int32_t slidingRotaryDim = {}

Rotary dimension for sliding attention RoPE.

int32_t fullRotaryDim = {}

Rotary dimension for full attention RoPE.

bool useContextDependentRope = {false}

Use context-dependent RoPE.

int32_t numDeepstackFeatures = {0}

Deepstack features (Qwen3-VL/Qwen3-Omni)

bool pleEnabled = {false}

Gemma4 PLE runtime preprocessor enabled.

int32_t numPleInputs = {0}

Number of PLE tensors bound into the engine (0 = disabled)

int32_t pleHiddenSize = {0}

Hidden dimension of each PLE tensor (0 = disabled)

int32_t maxSupportedLoraRank = {0}

Maximum LoRA rank (0 = no LoRA)

int32_t imageTokenId = {-1}

Special token ID for image (-1 = unused)

int32_t audioTokenId = {-1}

Special token ID for audio (-1 = unused)

std::vector<int32_t> eosTokenIds = {}

Additional EOS token IDs parsed from config.json eos_token_id array. Models like Gemma4 have multiple EOS tokens (e.g. [1, 106] for <eos> and <turn|>). Empty if eos_token_id is absent or scalar.

int32_t numLinearAttnLayers = {0}

Number of linear attention / recurrent layers.

int32_t recurrentStateNumHeads = {0}

Recurrent state heads (hv for GDN, mamba_num_heads for Mamba)

int32_t recurrentStateHeadDim = {0}

Recurrent state head dimension.

int32_t recurrentStateSize = {0}

Recurrent state dimension (v for GDN, dstate for Mamba)

int32_t convDim = {0}

Conv1d channel dimension.

int32_t convKernel = {0}

Conv1d kernel width.

int32_t maxVerifyTreeSize = {0}

Max seq_len the base engine accepts for proposal verification. Parsed from builder_config.max_verify_tree_size when isSpecDecodeBase == true; 0 otherwise. Consumers prefer the consolidated DeploymentConfig::specDecode when the deployment view is available.

int32_t maxDraftTreeSize = {0}

Max seq_len the draft engine accepts for proposal / draft generation. Parsed from builder_config.max_draft_tree_size by parseDraftEngineConfig; 0 on base / vanilla engines. Consumers prefer the consolidated DeploymentConfig::specDecode when the deployment view is available.

int32_t baseModelHiddenSize = {0}

Hidden dim the draft engine expects for its hidden_states_input binding (== the base engine’s hidden-state output dim as seen by the draft). Parsed from top-level base_model_hidden_size by parseDraftEngineConfig; left at 0 on base / vanilla engines. Differs from base.hiddenSize for EAGLE-3 (= base.hiddenSize * 3, multi-layer concat) and equals base.hiddenSize for MTP. The deployment factory copies this into DeploymentConfig::specDecode->baseOutputHiddenDim.

int32_t dflashBlockSize = {0}

DFlash draft block size. Parsed from dflash_config.block_size or top-level block_size on DFlash base/draft configs.

int32_t dflashMaskTokenId = {0}

DFlash mask token ID used to seed draft input blocks.

std::vector<int32_t> dflashTargetLayerIds = {}

Target decoder-layer IDs whose hidden states are concatenated for DFlash.

std::vector<HybridCacheManager::LayerType> layerTypes = {}

Absolute decoder-layer -> attention|mamba. Populated either from the canonical layer_types field in config.json or by broadcasting scalar numAttentionLayers/numLinearAttnLayers for back-compat. Size equals the number of stateful decoder layers (mlp/moe excluded on the Python side).

std::vector<KVLayerConfig> kvLayerConfigs = {}

Per-attention-layer KV config. Size equals the attention count in layerTypes. Indexed by LOCAL attention-layer index (0..numAttn-1), NOT absolute decoder-layer index.

std::vector<int32_t> kvSharingDonors = {}

Per-attention-layer KV sharing donor index. Size equals the attention count in layerTypes. Value of -1 means the layer owns its own KV cache (normal). A value >= 0 means this layer shares the KV cache from the donor attention layer at that LOCAL index (the donor’s cache is bound to this layer’s KV input). Used for Gemma4’s KV sharing where the last N layers reuse a donor’s cache.