Binding Names

namespace trt_edgellm

Argument bundles for the CuTe DSL FMHA launchers.

Each public run* entry point fills one of these once from its own arguments and members, and the launcher then spreads it across the generated descriptors. One struct per wrapper shape rather than a single superset: a superset would let a field that matters on one path (kvCacheCapacity on the dense path, tokensPerPage on the paged one) sit silently at zero on another.

These carry no generated types, so this header is safe to include from translation units that use the required FMHA-v2 runner, the optional CUTE_DSL_FMHA_BLACKWELL_ENABLED runner, or both. The descriptor-filling machinery itself lives in cuteDslTensorDescriptors.h, which stays free of any FMHA concept.

Helpers for populating the tensor descriptors emitted by the CuTe DSL C exporter.

Every AOT variant exports its own nominally distinct but layout-identical descriptor structs (fmha_d64_Tensor_q_tensor_t vs fmha_d128_Tensor_q_tensor_t) plus a cute_dsl_<variant>_wrapper entry point. There is no umbrella C type, so descriptor types are recovered here from the signature of the wrapper that consumes them: a call site names only the wrapper and the kernel module, and pairing a descriptor with the wrong variant is not expressible.

The exporter (cutlass/cute/export/c_header_generator.py) always names the members data, dynamic_shapes and dynamic_strides, but emits each array only when its dynamic mask is non-empty. A rank-1 descriptor therefore has no dynamic_strides member at all and needs makeCuSeqLenTensor() rather than the strided builders below.

namespace binding_names

Core LLM Input/Output Bindings

char const *kInputsEmbeds = "inputs_embeds"

Input embeddings tensor - contains the embedded input sequence.

Shape: [batch_size, sequence_length, hidden_size] (FLOAT16)

char const *kPleTokenEmbedsTemplate = "ple_token_embeds"

Gemma4 per-layer token-identity embedding input template.

Template: “ple_token_embeds_{layer_idx}” Shape: [batch_size, sequence_length, ple_hidden_size] (FLOAT16/BFLOAT16)

char const *kContextLengths = "context_lengths"

Context lengths tensor - specifies the actual length of each sequence in the batch.

Shape: [batch_size] (INT32)

char const *kLastTokenIds = "last_token_ids"

Last token IDs tensor - indices of the last tokens to extract from hidden states.

Shape: [batch_size] for Eagle models, [batch_size, 1] for vanilla models (INT64)

char const *kPhaseIsEncoder = "phase_is_encoder"

DiffusionGemma phase selector. Non-zero selects encoder-phase layer scalars.

char const *kSelectTokenIndices = "select_token_indices"

DiffusionGemma hidden-state gather indices for logits output.

char const *kContextMaskSelector = "context_mask_selector"

DiffusionGemma context-mask shape selector: [0] causal, [B] padding/non-causal.

char const *kCanvasIds = "canvas_ids"

DiffusionGemma fused conditioning engine canvas token IDs.

char const *kPrevSelfConditioningEmbeds = "prev_self_conditioning_embeds"

DiffusionGemma fused conditioning engine previous hidden feedback embeddings.

char const *kNextSelfConditioningEmbeds = "next_self_conditioning_embeds"

DiffusionGemma fused conditioning engine next hidden feedback embeddings.

char const *kSelfConditioningTemperature = "self_conditioning_temperature"

DiffusionGemma fused conditioning engine feedback softmax temperature.

char const *kLogits = "logits"

Output logits tensor - probability distribution over vocabulary.

Shape: [batch_size, vocab_size] or [select_tokens, vocab_size] (FLOAT32)

char const *kOutputHiddenStates = "hidden_states"

Output hidden states tensor - intermediate representations for speculative decoding.

Shape: [batch_size, sequence_length, hidden_dim] (FLOAT16)

char const *kDFlashTargetHiddenConcat = "dflash_target_hidden_concat"

Cached speculative draft model input: concatenated target hidden states.

Shape: [batch_size, context_length, base_output_hidden_dim] (FLOAT16)

char const *kDFlashDeltaLengths = "dflash_delta_lengths"

Cached speculative draft model input: per-batch delta lengths for multi-batch.

Shape: [batch_size] (INT32)

char const *kDSparkHiddenStates = "dspark_hidden_states"

DSpark draft model output: final proposal hidden states.

Shape: [batch_size, proposal_length, hidden_size] (FLOAT16)

char const *kDSparkHeadsFileName = "dspark_heads.safetensors"

DSpark Markov/confidence head sidecar file names.

char const *kDSparkHeadsInfoFileName = "dspark_heads_info.json"

char const *kPleEmbeddingFileName = "ple_embedding.safetensors"

Gemma4 PLE token-identity embedding table sidecar.

Contains tensor “weight” with shape [vocab_size_per_layer_input, num_ple_inputs * ple_hidden_size].

Positional Encoding Bindings

char const *kRopeCosSin = "rope_rotary_cos_sin"

Rotary positional encoding cos/sin cache tensor.

Shape: [batch_size, max_seq_len, rotary_dim] (FLOAT32)

char const *kRopeCosSinSliding = "rope_rotary_cos_sin_sliding"

Rotary positional encoding cos/sin cache tensor for sliding attention layers.

Shape: [batch_size, max_seq_len, sliding_rotary_dim] (FLOAT32)

char const *kRopeCosSinFull = "rope_rotary_cos_sin_full"

Rotary positional encoding cos/sin cache tensor for full attention layers.

Shape: [batch_size, max_seq_len, full_rotary_dim] (FLOAT32)

KV Cache Bindings

char const *kKVCacheStartIndex = "kvcache_start_index"

KV cache start index tensor - starting position for KV cache reuse.

Shape: [batch_size] (INT32)

char const *kKVPageTable = "kv_page_table"

KV page-table tensor - per-request page indices into the paged two-pool KV cache.

K page ids in row 0, derived V page ids (K + numPages) in row 1.

Shape: [batch_size, 2, max_pages_per_seq] (INT32)

char const *kPastKeyValuesTemplate = "past_key_values"

Past key-value cache tensor template - use with layer index formatting.

Template: “past_key_values_{layer_idx}” Shape: [2, num_pages, kTOKENS_PER_PAGE, num_kv_heads, head_dim] paged pool (FLOAT16/FP8)

char const *kPresentKeyValuesTemplate = "present_key_values"

Present key-value cache tensor template - use with layer index formatting.

Template: “present_key_values_{layer_idx}” Shape: [2, num_pages, kTOKENS_PER_PAGE, num_kv_heads, head_dim] paged pool, aliases past (FLOAT16/FP8)

char const *kKCacheTemplate = "k_cache"

K cache tensor template for TensorRT native KVCacehUpdate operations - use with layer index formatting.

Template: “k_cache_{layer_idx}” Shape: [batch_size, num_kv_heads, seq_len, head_dim] (FLOAT16)

char const *kVCacheTemplate = "v_cache"

V cache tensor template for TensorRT native KVCacheUpdate operations - use with layer index formatting.

Template: “v_cache_{layer_idx}” Shape: [batch_size, num_kv_heads, seq_len, head_dim] (FLOAT16)

char const *kPresentKCacheTemplate = "present_k_cache"

Present K cache tensor template for TensorRT native KVCacheUpdate operations - use with layer index formatting.

Template: “present_k_cache_{layer_idx}” Shape: [batch_size, num_kv_heads, seq_len, head_dim] (FLOAT16)

char const *kPresentVCacheTemplate = "present_v_cache"

Present V cache tensor template for TensorRT native KVCacheUpdate operations - use with layer index formatting.

Template: “present_v_cache_{layer_idx}” Shape: [batch_size, num_kv_heads, seq_len, head_dim] (FLOAT16)

Alpamayo 1 action head input/output bindings

char const *kNoiseTrajectory = "noise_trajectory"

Noise trajectory input for one denoising step.

Shape: [batch_size, num_waypoints, 2] (FLOAT32)

char const *kTimeStepsT0 = "time_steps_t0"

Denoising time step t0 (start)

Shape: [1] (FLOAT32)

char const *kTimeStepsT1 = "time_steps_t1"

Denoising time step t1 (end)

Shape: [1] (FLOAT32)

char const *kDenoisedTrajectory = "denoised_trajectory"

Denoised trajectory output from one denoising step.

Shape: [batch_size, num_waypoints, 2] (FLOAT32)

Recurrent State Bindings (Mamba / GDN / linear-attention layers)

char const *kRecurrentStateTemplate = "recurrent_state"

Past recurrent state tensor template.

Template: “recurrent_state_{recurrent_layer_idx}” Shape: [batch_size, recurrentNumHeads, recurrentHeadDim, recurrentStateSize]

char const *kPresentRecurrentStateTemplate = "present_recurrent_state"

Present recurrent state tensor template.

Template: “present_recurrent_state_{recurrent_layer_idx}” Shape: [batch_size, recurrentNumHeads, recurrentHeadDim, recurrentStateSize]

char const *kConvStateTemplate = "conv_state"

Past conv state tensor template for recurrent layers.

Template: “conv_state_{recurrent_layer_idx}” Shape: [batch_size, conv_dim, conv_kernel_size] (FLOAT16)

char const *kPresentConvStateTemplate = "present_conv_state"

Present conv state tensor template for recurrent layers.

Template: “present_conv_state_{recurrent_layer_idx}” Shape: [batch_size, conv_dim, conv_kernel_size] (FLOAT16)

char const *kIntermediateConvStateTemplate = "intermediate_conv_state"

Intermediate conv state output template for MTP speculative decoding.

Template: “intermediate_conv_state_{recurrent_layer_idx}” Shape: [batch_size, seq_len, conv_dim, conv_kernel_size] (FLOAT16)

char const *kIntermediateRecurrentStateTemplate = "intermediate_recurrent_state"

Intermediate recurrent state output template for MTP speculative decoding.

Template: “intermediate_recurrent_state_{recurrent_layer_idx}” Shape: [batch_size, seq_len, recurrentNumHeads, recurrentHeadDim, recurrentStateSize] (FLOAT32)

char const *kReplayDaStateTemplate = "replay_da_state"

Mamba spec-verify replay-stash output templates (FP32). Instead of a per-token full-state snapshot, the Mamba plugin stashes the minimal per-token replay inputs; the runtime reconstructs the accepted recurrent state from them after verification. dA: [batch, seq_len, recurrentNumHeads] u: [batch, seq_len, recurrentNumHeads, recurrentHeadDim] B: [batch, seq_len, recurrentNumGroups, recurrentStateSize].

char const *kReplayUStateTemplate = "replay_u_state"

char const *kReplayBStateTemplate = "replay_b_state"

Eagle Speculative Decoding Bindings

char const *kBaseModelHiddenStates = "hidden_states_input"

Base model hidden states input for Eagle draft models.

Shape: [batch_size, sequence_length, base_hidden_dim] (FLOAT16)

char const *kDraftModelHiddenStates = "hidden_states_from_draft"

Draft model hidden states input for Eagle draft models.

Shape: [batch_size, sequence_length, draft_hidden_dim] (FLOAT16)

char const *kAttentionMask = "attention_mask"

Attention mask for Eagle models - packed tree attention mask.

Shape: [batch_size, tree_size, packed_mask_len] (INT32 for base, INT8 for draft)

char const *kVisionBlockIds = "vision_block_ids"

Gemma4 Unified prefill block IDs: [batch_size, sequence_length] INT32.

char const *kAttentionPosId = "attention_pos_id"

Attention position IDs for Eagle models.

Shape: [batch_size, tree_size] (INT32)

char const *kSpecVerifyPhaseMarker = "spec_verify_phase_marker"

Shape-only marker for speculative verification in hybrid MTP/DFlash base engines.

Shape: [0] for normal prefill/decode/reset, [1] for MTP/DFlash verification (INT32). The INT32 payload is ignored. Hybrid GDN/conv plugins only need to distinguish speculative verify seq_len > 1 from ordinary prefill seq_len > 1.

char const *kSkipSoftmaxScale = "skip_softmax_scale"

Runtime skip-softmax override carrier (optional AttentionPlugin input).

1-D INT8 dummy whose SHAPE (dims[0] = integer scale factor S) conveys the override; 0 keeps the engine-carried calibrated default.

char const *kTreeParentIds = "tree_parent_ids"

DDTree parent node ids for hybrid DFlash base verification.

Shape: [batch_size, verify_tree_size] (INT32). Each entry points to the flattened parent node whose hybrid state is used to evaluate the current node.

char const *kTreeDepths = "tree_depths"

DDTree node depths for hybrid DFlash base verification.

Shape: [batch_size, verify_tree_size] (INT32). Depth is used for positional metadata and for tree-state kernels that need node order information.

Visual Encoder Bindings (Qwen-VL, InternVL)

char const *kVisualInput = "input"

Visual input tensor for vision transformers.

Shape: [sequence_length, input_dim] for Qwen-VL, [num_blocks, channels, height, width] for InternVL

char const *kVisualShuffleIndices = "shuffle_indices"

Pixel-shuffle gather indices for the Nemotron-Omni visual engine.

Shape: [num_out_tokens, scale^2] (INT64). Grid-dependent, computed by the runtime so one engine serves square image tiles and non-square video grids.

char const *kVisualOutput = "output"

Visual output tensor from vision transformers.

Shape: [num_image_tokens, hidden_size] (FLOAT16)

char const *kRotaryPosEmb = "rotary_pos_emb"

Rotary positional embeddings for visual inputs.

Shape: [sequence_length, embed_dim] (FLOAT32)

char const *kPixelPositionIds = "pixel_position_ids"

Gemma4 pixel position ids for visual patch embeddings and 2-D RoPE.

Shape: [sequence_length, 2] (INT64)

char const *kCuSeqlens = "cu_seqlens"

Cumulative sequence lengths for ragged ViT attention.

Shape: [num_images + 1] (INT32)

char const *kKvLengths = "kv_lengths"

KV sequence lengths for TRT-native attention (TRT >= 11).

Same data as cu_seqlens but must be a separate tensor — TRT IAttentionV2 requires distinct tensors for query_lengths and kv_lengths inputs.

Shape: [num_images + 1] (INT32)

char const *kKvLengthsWindow = "kv_lengths_window"

Window KV sequence lengths for Qwen2.5-VL TRT-native attention.

Same data as cu_window_seqlens but must be a separate tensor for TRT_Attention query_lengths/kv_lengths inputs in window-attention blocks.

Shape: [num_windows + 1] (INT32)

char const *kMaxSeqLenCarrier = "max_seqlen_carrier"

Shape-only input used to convey runtime max sequence-length for FMHA launch.

Shape: [max_seqlen] (INT32)

char const *kPoolingWeights = "pooling_weights"

Gemma4 position-aware pooling weights.

Shape: [num_image_tokens, sequence_length] (FLOAT16)

char const *kCuWindowSeqlens = "cu_window_seqlens"

Cumulative window sequence lengths for Qwen2.5-VL window attention.

Shape: [num_windows + 1] (INT32)

char const *kWindowIndex = "window_index"

Window index for Qwen2.5-VL sliding window attention.

Shape: [num_windows] (INT64)

char const *kReverseWindowIndex = "reverse_window_index"

Reverse window index for Qwen2.5-VL sliding window attention.

Shape: [num_windows] (INT64)

char const *kFastPosEmbIdx = "fast_pos_embed_idx"

Fast position embeddings index tensor for Qwen3-VL vision model.

Shape: [4, sequence_length] (INT64)

char const *kFastPosEmbWeight = "fast_pos_embed_weight"

Fast position embeddings weight tensor for Qwen3-VL vision model.

Shape: [4, sequence_length] (FLOAT16)

char const *kDeepstackFeaturesTemplate = "deepstack_features"

Deepstack features tensor for Qwen3-VL vision model (visual encoder output)

Shape: [num_image_tokens, hidden_size] (FLOAT16)

char const *kDeepstackEmbedsTemplate = "deepstack_embeds"

Deepstack embeddings tensor template for Qwen3-VL text model (LLM input)

Template: “deepstack_embeds_{layer_idx}” where layer_idx is 0, 1, or 2 Shape: [batch_size, sequence_length, hidden_size] (FLOAT16)

Vocabulary Mapping Configuration

char const *kReducedVocabSizeKey = "reduced_vocab_size"

JSON configuration key for reduced vocabulary size.

Used to check if the model uses vocabulary reduction optimization

char const *kVocabMapFileName = "vocab_map.safetensors"

Vocabulary mapping file name.

SafeTensors file containing mapping between full and reduced vocabulary

char const *kDraftVocabMapFileName = "draft_vocab_map.safetensors"

Draft model vocabulary mapping file name.

SafeTensors file containing mapping from reduced draft vocabulary to full vocabulary.

Note

Currently consumed only by DFlashDecoder (gated on the draft engine config’s reduced_vocab_size > 0) and produced only by the DFlash draft export path. If a future decoder adopts draft vocab reduction, it must (a) gate the load on the same config field and (b) be added as a consumer of this constant.

Audio Encoder Bindings (Qwen3-Omni)

char const *kAudioInputFeatures = "input_features"

Raw framed PCM input used by encoder-free audio embedders.

Shape: [1, num_frames, frame_size] (FLOAT16)

char const *kAudioPaddedFeatures = "padded_feature"

Audio padded features tensor - chunked and padded Mel-spectrogram.

Shape: [num_chunks, mel_bins, max_chunk_len] (FLOAT16)

char const *kAudioPaddedMaskIndices = "padded_mask_after_cnn_indices"

Audio padded mask indices - nonzero indices from mask.

Shape: [num_valid_elements, 2] (INT64) Each row is [chunk_idx, position_idx] indicating valid positions after CNN downsampling

char const *kAudioAttentionMask = "attention_mask"

Audio attention mask - block-diagonal mask for chunk-wise attention.

Shape: [num_attention_elems, num_attention_elems] (FLOAT16) Block-diagonal matrix where each block corresponds to one audio chunk

char const *kAudioOutput = "last_hidden_state"

Audio encoder output - audio embeddings.

Shape: [num_audio_tokens, hidden_size] (FLOAT16)

CodePredictor Bindings (Qwen3-Omni)

char const *kLmHeadWeight = "lm_head_weight"

LM head weight tensor - dynamically bound weight for CodePredictor.

Shape: [vocab_size, hidden_size] (FLOAT16) This is used for dynamic lm_head selection in CodePredictor (15 different heads for RVQ layers)

char const *kLmHeads = "lm_heads"

Stacked CodePredictor lm_heads + device-selected head index.

lm_heads: [num_heads, vocab_size, hidden_size] (FLOAT16), lm_head_idx: [1] (INT32). The head is gathered inside the engine, so bindings stay step-invariant and one CUDA graph serves every decode step.

char const *kLmHeadIdx = "lm_head_idx"

Code2Wav Vocoder Bindings (Qwen3-Omni)

char const *kCode2WavCodes = "codes"

Code2Wav input codes tensor - RVQ codec codes for vocoder.

Shape: [batch_size, num_quantizers, sequence_length] (INT32) num_quantizers: 15 for Qwen3-Omni

char const *kCode2WavWaveform = "waveform"

Code2Wav output waveform tensor - generated audio waveform.

Shape: [batch_size, 1, waveform_length] (FLOAT32) Values in range [-1.0, 1.0]

LoRA (Low-Rank Adaptation) Bindings

char const *kLoraAPrefix = "lora_A"

LoRA A weight matrix prefix - use with layer/component specific suffixes.

Template: “lora_A_{component}_{layer}” Shape: [gemm_k, lora_rank] (FLOAT16)

char const *kLoraBPrefix = "lora_B"

LoRA B weight matrix prefix - use with layer/component specific suffixes.

Template: “lora_B_{component}_{layer}” Shape: [lora_rank, gemm_n] (FLOAT16)

char const *kEdgellmVersion = "edgellm_version"

EDGELLM version.

Value: “major.minor.patch.build” Example: “0.5.0.0”

Utility Functions

inline std::string formatKVCacheName(

int32_t layerIdx,

bool isPast = true

)

Format KV cache binding name for a specific layer.

Parameters:

• layerIdx – The decoder layer index

• isPast – Whether this is past (true) or present (false) key-values

Returns:

Formatted binding name like “past_key_values_0” or “present_key_values_0”

inline std::string formatKCacheName(

int32_t layerIdx,

bool isPast = true

)

Format K cache binding name for a specific layer (TensorRT native operations)

Parameters:

• layerIdx – The decoder layer index

• isPast – Whether this is past (true) or present (false) K cache

Returns:

Formatted binding name like “k_cache_0” or “present_k_cache_0”

inline std::string formatVCacheName(

int32_t layerIdx,

bool isPast = true

)

Format V cache binding name for a specific layer (TensorRT native operations)

Parameters:

• layerIdx – The decoder layer index

• isPast – Whether this is past (true) or present (false) V cache

Returns:

Formatted binding name like “v_cache_0” or “present_v_cache_0”

inline std::string formatRecurrentStateName(

int32_t recurrentLayerIdx,

bool isPast = true

)

Format recurrent state binding name for a specific recurrent layer.

Parameters:

• recurrentLayerIdx – The recurrent layer index (0-based, only counting recurrent layers)

• isPast – Whether this is past (true) or present (false) recurrent state

Returns:

Formatted binding name like “recurrent_state_0” or “present_recurrent_state_0”

inline bool isRecurrentStateBinding(std::string const &bindingName)

Check if a binding name is a recurrent state tensor.

Parameters:

bindingName – The tensor binding name to check

Returns:

True if the binding is a recurrent state tensor

inline std::string formatConvStateName(

int32_t recurrentLayerIdx,

bool isPast = true

)

Format conv state binding name for a specific recurrent layer.

Parameters:

• recurrentLayerIdx – The recurrent layer index (0-based, only counting recurrent layers)

• isPast – Whether this is past (true) or present (false) conv state

Returns:

Formatted binding name like “conv_state_0” or “present_conv_state_0”

inline bool isConvStateBinding(std::string const &bindingName)

Check if a binding name is a conv state tensor.

Parameters:

bindingName – The tensor binding name to check

Returns:

True if the binding is a conv state tensor

inline std::string formatIntermediateRecurrentStateName(

int32_t recurrentLayerIdx

)

Format intermediate recurrent state binding name for MTP.

Parameters:

recurrentLayerIdx – The recurrent layer index (0-based)

Returns:

Formatted binding name like “intermediate_recurrent_state_0”

inline std::string formatIntermediateConvStateName(

int32_t recurrentLayerIdx

)

Format intermediate conv state binding name for MTP.

Parameters:

recurrentLayerIdx – The recurrent layer index (0-based)

Returns:

Formatted binding name like “intermediate_conv_state_0”

inline std::string formatReplayDaStateName(int32_t recurrentLayerIdx)

Format the Mamba spec-verify replay-stash binding names (dA / u / B).

Parameters:

recurrentLayerIdx – The recurrent layer index (0-based)

inline std::string formatReplayUStateName(int32_t recurrentLayerIdx)

inline std::string formatReplayBStateName(int32_t recurrentLayerIdx)

inline bool isLoraBinding(std::string const &bindingName) noexcept

Check if a binding name is a LoRA weight tensor.

Parameters:

bindingName – The tensor binding name to check

Returns:

True if the binding is a LoRA weight tensor

inline bool isKVCacheBinding(std::string const &bindingName) noexcept

Check if a binding name is a KV cache tensor.

Parameters:

bindingName – The tensor binding name to check

Returns:

True if the binding is a KV cache tensor

inline std::string formatDeepstackFeaturesName(int32_t layerIdx)

Format deepstack features binding name for a specific layer.

Parameters:

layerIdx – The layer index

Returns:

Formatted binding name like “deepstack_features_0”

inline std::string formatDeepstackEmbedsName(int32_t embedIdx)

Format deepstack embeddings binding name for a specific index.

Parameters:

embedIdx – The embedding index (0, 1, or 2 for Qwen3VL)

Returns:

Formatted binding name like “deepstack_embeds_0”

inline std::string formatPleTokenEmbedsName(int32_t layerIdx)

Format Gemma4 PLE token embedding input binding name.

Parameters:

layerIdx – The decoder layer index

Returns:

Formatted binding name like “ple_token_embeds_0”

namespace binding_names

Unified tensor binding names for TensorRT engines.

This namespace provides a centralized location for all tensor binding names used across both the builder and runtime components to ensure consistency and avoid duplication.