Cute Dsl FMHA Runner#
-
class CuteDslFMHARunner#
Unified runner for CuTe DSL compiled FMHA kernels (Blackwell SM100/101/103/110).
Supports two execution modes via AOT-compiled kernel variants:
LLM prefill/chunked-prefill: batched Q [B,S_q,H_q,D] + combined KV cache [B,2,H_kv,Cap,D] with causal masking and optional sliding window.
ViT: packed varlen separate Q/K/V [total_S,H,D] with cu_seqlens [B+1] for ragged batching, bidirectional attention (no causal mask).
Each mode has its own kernel modules and run() overload.
Public Functions
- CuteDslFMHARunner(
- int32_t numQHeads,
- int32_t numKVHeads,
- int32_t headDim,
- int32_t batchSize = 0,
- int32_t seqLenQ = 0,
- int32_t kvCacheCapacity = 0
-
~CuteDslFMHARunner() = default#
-
CuteDslFMHARunner(CuteDslFMHARunner const&) = delete#
-
CuteDslFMHARunner &operator=(CuteDslFMHARunner const&) = delete#
- bool preflightLlm(
- cudaStream_t stream,
- int32_t slidingWindowSize = INT_MAX,
- bool fp8Input = false,
- int32_t const *skipSoftmaxScaleTable = nullptr,
- int32_t skipSoftmaxScaleTableLen = 0
Ensures the exact dense LLM variant selected by run() is loaded.
- bool preflightPaged(
- cudaStream_t stream,
- int32_t slidingWindowSize = INT_MAX,
- bool fp8Input = false,
- bool isCausal = true,
- int32_t const *skipSoftmaxScaleTable = nullptr,
- int32_t skipSoftmaxScaleTableLen = 0,
- bool useBidirectional = false
Ensures the exact paged LLM variant selected by runPaged() is loaded.
- bool preflightPagedRagged(
- cudaStream_t stream,
- int32_t slidingWindowSize = INT_MAX,
- trt_edgellm::DataType inputDataType = trt_edgellm::DataType::kHALF,
- bool useBidirectional = false
Ensures the exact ragged paged LLM variant selected by runPagedRagged() is loaded.
-
bool preflightViT(cudaStream_t stream)#
Ensures the exact packed ViT variant selected by run() is loaded.
- bool run(
- void const *qPtr,
- void const *kvPtr,
- void *oPtr,
- int32_t const *cuKVSeqLens,
- cudaStream_t stream,
- float attentionScale,
- int32_t slidingWindowSize = INT_MAX,
- bool fp8Input = false,
- float qScale = 1.0F,
- float kScale = 1.0F,
- float vScale = 1.0F,
- int32_t const *skipSoftmaxScaleTable = nullptr,
- int32_t skipSoftmaxScaleTableLen = 0,
- int32_t const *skipSoftmaxCuKVSeqLens = nullptr
LLM FMHA: batched Q + combined KV cache with causal masking.
Output is always FP16. Selects kernel variant based on fp8Input:
fp8Input=false → FP16 kernels (all scales ignored)
fp8Input=true → FP8-input / FP16-output kernels
- Parameters:
qPtr – Query [B, S_q, H_q, D]
kvPtr – Combined KV cache [B, 2, H_kv, Cap, D]
oPtr – Output [B, S_q, H_q, D] (always FP16)
cuKVSeqLens – Cumulative KV sequence lengths [B+1]
stream – CUDA stream
attentionScale – Model-defined multiplier applied to QK^T before softmax. For FP8 input, the effective softmax scale is attentionScale * qScale * kScale.
slidingWindowSize – Sliding window size (INT_MAX = disabled)
fp8Input – Whether Q/KV are FP8 E4M3
qScale – Q dequant scale (quant→orig), ignored when fp8Input=false
kScale – K dequant scale (quant→orig), ignored when fp8Input=false
vScale – V dequant scale (quant→orig), applied to the attention output and ignored when fp8Input=false
skipSoftmaxScaleTable – Skip-softmax (BLASST) scale table, device Int32 of flattened (seqlen_bin, S) pairs with bins ascending; a flat S is the one-row table {0, S}. When bound the skip-softmax variant is dispatched (FP16 causal only): per sequence the kernel floor-looks-up S by its KV length, derives the threshold log2(S / seqlen_kv) and skips the P*V GEMM of KV tiles whose contribution is negligible — approximate. Sequences below the first bin or with S >= seqlen_kv skip nothing (dense result to fp16 rounding).
skipSoftmaxScaleTableLen – Element count of the table; 0 (the default) dispatches the dense cubin.
skipSoftmaxCuKVSeqLens – Actual cumulative KV lengths [B + 1] for the lookup when cuKVSeqLens is padded to the batch’s longest query (bottom-right alignment); nullptr = cuKVSeqLens is already actual.
- bool runPaged(
- void const *qPtr,
- void const *pagedKVPoolPtr,
- int32_t const *kvCachePageList,
- void *oPtr,
- int32_t const *cuKVSeqLens,
- int32_t numPages,
- int32_t maxPagesPerSeq,
- int32_t tokensPerPage,
- trt_edgellm::DataType kvDataType,
- cudaStream_t stream,
- float attentionScale,
- int32_t slidingWindowSize = INT_MAX,
- bool fp8Input = false,
- float qScale = 1.0F,
- float kScale = 1.0F,
- float vScale = 1.0F,
- bool isCausal = true,
- int32_t const *skipSoftmaxScaleTable = nullptr,
- int32_t skipSoftmaxScaleTableLen = 0,
- int32_t const *skipSoftmaxCuKVSeqLens = nullptr,
- int32_t const *bidirectionalBlockBegin = nullptr,
- int32_t const *bidirectionalBlockEnd = nullptr
LLM FMHA over a paged KV cache.
Dispatches a dedicated CuTe DSL AOT variant that reads K/V directly from a paged pool using kvCachePageList. The logical descriptor shape is [numPages, H_kv, tokensPerPage, D], while the physical pool is fixed to NHD [numPages, tokensPerPage, H_kv, D]. This path maps one logical K/V TMA tile to one physical page. The current CuTe DSL variants use a K/V tile width of 128 tokens, so tokensPerPage must be 128 to avoid multi-page tile stitching or a gather workspace. Setting both block range pointers selects the FP16 D512 bidirectional-mask variant; mixed null pointers are invalid.
- Parameters:
qPtr – Query [B, S_q, H_q, D]
pagedKVPoolPtr – Paged KV pool [numPages, tokensPerPage, H_kv, D]
kvCachePageList – Page table [B, 2, maxPagesPerSeq], K pages then V pages
oPtr – Output [B, S_q, H_q, D] (always FP16)
cuKVSeqLens – Cumulative KV sequence lengths [B+1]
numPages – Number of pages in the paged KV pool
maxPagesPerSeq – Max logical pages per sequence
tokensPerPage – Number of tokens per page
kvDataType – Paged KV cache dtype (FP16 or FP8)
stream – CUDA stream
slidingWindowSize – Sliding window size (INT_MAX = disabled)
fp8Input – Whether Q/KV are FP8 E4M3
qScale – Q dequant scale, ignored when fp8Input=false
kScale – K dequant scale, ignored when fp8Input=false
vScale – V dequant scale, ignored when fp8Input=false
isCausal – Whether to dispatch a causal or dense non-causal variant
skipSoftmaxScaleTable – Skip-softmax scale table (see run()); nullptr / length 0 = dense cubin
skipSoftmaxScaleTableLen – Element count of skipSoftmaxScaleTable
skipSoftmaxCuKVSeqLens – Actual cumulative KV lengths [B + 1] (see run()); nullptr = cuKVSeqLens
bidirectionalBlockBegin – Optional inclusive bidirectional-block begin positions [B, S_q]. Text/padding rows use -1; every row in a disjoint contiguous vision run must repeat that run’s begin position.
bidirectionalBlockEnd – Optional inclusive bidirectional-block end positions [B, S_q]. Text/padding rows use -1; every row in a disjoint contiguous vision run must repeat that run’s end position.
- bool runPagedRagged(
- void const *qPtr,
- void const *pagedKVPoolPtr,
- int32_t const *kvCachePageList,
- void *oPtr,
- int32_t const *cuQSeqLens,
- int32_t const *cuKVSeqLens,
- int32_t totalQSeqLen,
- int32_t maxQSeqLen,
- int32_t numFlatPages,
- int32_t maxPagesPerSeq,
- int32_t tokensPerPage,
- cudaStream_t stream,
- float attentionScale,
- int32_t slidingWindowSize = INT_MAX,
- trt_edgellm::DataType inputDataType = trt_edgellm::DataType::kHALF,
- float qScale = 1.0F,
- float kScale = 1.0F,
- float vScale = 1.0F,
- float const *attentionSinks = nullptr,
- int32_t const *bidirectionalBlockBegin = nullptr,
- int32_t const *bidirectionalBlockEnd = nullptr
Runs causal or sliding-causal FMHA with optional vision blocks over packed Q/O and native paged KV.
cuQSeqLens and cuKVSeqLens are monotonic prefix arrays of length batchSize + 1. cuQSeqLens starts at zero and ends at totalQSeqLen. cuKVSeqLens contains the complete logical KV history, including prior cached tokens. maxQSeqLen is the maximum adjacent difference in cuQSeqLens, not the packed buffer allocation capacity.
- Parameters:
qPtr – Query [totalQSeqLen, H_q, D]
pagedKVPoolPtr – Flattened K/V pool [numFlatPages, tokensPerPage, H_kv, D]
kvCachePageList – Page table [B, 2, maxPagesPerSeq]
oPtr – Output [totalQSeqLen, H_q, D] in FP16
cuQSeqLens – Cumulative Q sequence lengths [B+1]
cuKVSeqLens – Cumulative logical KV sequence lengths [B+1]
totalQSeqLen – Number of active packed Q/O tokens
maxQSeqLen – Maximum logical Q sequence length in the batch
numFlatPages – Number of pages in the flattened K/V pool
maxPagesPerSeq – Maximum logical pages per sequence
tokensPerPage – Number of tokens per page
stream – CUDA stream
attentionScale – Model-defined multiplier applied to QK^T before softmax
slidingWindowSize – Sliding window size, or INT_MAX when disabled
inputDataType – Q/KV input dtype; FP16 and FP8 E4M3 are supported
qScale – Q dequant scale, ignored for FP16 input
kScale – K dequant scale, ignored for FP16 input
vScale – V dequant scale, ignored for FP16 input
bidirectionalBlockBegin – Optional device Int32 [totalQSeqLen] inclusive sequence-relative KV begins.
bidirectionalBlockEnd – Optional device Int32 [totalQSeqLen] inclusive sequence-relative KV ends. Both boundaries must be bound together (FP16 D512 only). Each image row repeats its contiguous image interval; text rows use -1. Row indices follow packed Q order, while values include cached KV history.
attentionSinks – Optional device FP32 logits [H_q], in the scaled attention logit domain
- bool run(
- void const *qPtr,
- void const *kPtr,
- void const *vPtr,
- void *oPtr,
- int32_t const *cuSeqLens,
- int32_t totalSeqLen,
- int32_t maxSeqLen,
- int32_t batchSize,
- cudaStream_t stream,
- float attentionScale,
- bool fp8Input = false,
- float qScale = 1.0F,
- float kScale = 1.0F,
- float vScale = 1.0F
ViT FMHA: packed varlen separate Q/K/V, bidirectional.
Output is always FP16. Selects kernel variant based on fp8Input:
fp8Input=false → FP16 kernels (all scales ignored)
fp8Input=true → FP8-input / FP16-output kernels
- Parameters:
qPtr – Query [total_S, H, D]
kPtr – Key [total_S, H, D]
vPtr – Value [total_S, H, D]
oPtr – Output [total_S, H, D] (always FP16)
cuSeqLens – Cumulative sequence lengths [B+1]
totalSeqLen – Sum of all sequence lengths
maxSeqLen – Longest individual sequence length
batchSize – Number of sequences
stream – CUDA stream
attentionScale – Absolute multiplier applied to QK^T before softmax. For FP8 input, the effective softmax scale is attentionScale * qScale * kScale.
fp8Input – Whether Q/K/V are FP8 E4M3
qScale – Q dequant scale (quant→orig), ignored when fp8Input=false
kScale – K dequant scale (quant→orig), ignored when fp8Input=false
vScale – V dequant scale (quant→orig), applied to the attention output and ignored when fp8Input=false
Public Static Functions
-
static bool canImplement(int32_t headSize, int32_t smVersion)#
- static bool canImplementPagedRagged(
- int32_t numQHeads,
- int32_t numKVHeads,
- int32_t headSize,
- int32_t smVersion,
- trt_edgellm::DataType inputDataType,
- CuteDslFMHAMaskType maskType
Packed-Q/O native paged attention supports optional FP32 per-query-head sinks for FP16 and FP8 input.
-
static bool canImplementViT(int32_t headSize, int32_t smVersion)#
- static bool shouldDispatchSkipSoftmax(
- int32_t const *skipSoftmaxScaleTable,
- int32_t skipSoftmaxScaleTableLen
Dense/skip cubin dispatch predicate: the skip-softmax variant is dispatched only when a scale table is bound (at least one (seqlen_bin, S) pair). An empty table runs the dense cubin, which the runtime uses for batches that could skip nothing: the two variants are independently compiled cubins, so the skip kernel’s register/branch tax never reaches dense work.