Compact Phase Support

struct CompactPhaseSupport

Public Members

bool supported = {false}

char const *reason = {nullptr}

What is missing, as a static string; nullptr when supported.

struct AttentionLayerTraits

The static configuration of one AttentionPlugin layer that decides its kernels.

Public Members

int32_t numQHeads = {0}

Per rank.

int32_t numKVHeads = {0}

Per rank.

int32_t headSize = {0}

int32_t smVersion = {0}

The device SM, before or after thorRenumberedSm().

int32_t tpSize = {1}

DataType dataType = {DataType::kHALF}

bool fp8KVCache = {false}

bool slidingWindow = {false}

bool supportsBoundedKVCache = {false}

bool kvShared = {false}

bool qOnlyInput = {false}

The packed QKV input carries Q only (a KV-shared layer without the donor’s current K/V).

bool treeAttention = {false}

bool visionBlockAttention = {false}

bool attentionSink = {false}

bool contextMaskSelector = {false}

bool trt_edgellm::isCtmOnlySM(int32_t smVersion) noexcept

SMs whose attention decodes on the CTM kernels only. Holds for an SM before and after thorRenumberedSm().

int32_t trt_edgellm::thorRenumberedSm(int32_t smVersion) noexcept

Workaround for the CUDA 12/13 Thor SM renumbering: Thor (SM110) selects the SM101 attention kernels, which run on it. Idempotent.

bool trt_edgellm::ctmDecodeDisabledByEnv(int32_t smVersion) noexcept

Whether EDGELLM_DISABLE_CTM_DECODE turns CTM decode off on SM smVersion; CTM-only SMs ignore it. The variable is read once per process.

bool trt_edgellm::cuteDslArtifactCarriesSm(

int32_t smVersion

) noexcept

Whether the linked CuTe DSL artifact was built for SM smVersion (its metadata.json gpu_arch list). The kernel-group macros are defined per artifact, not per SM, so a predicate whose kernels come from the artifact checks this too.

bool trt_edgellm::decodesOnCtmOnly(

int32_t smVersion,

int32_t tpSize

) noexcept

Whether attention on SM smVersion at tensor-parallel size tpSize decodes on the CTM kernels only.

bool trt_edgellm::canUseCtmDecode(

AttentionLayerTraits const &layer

) noexcept

Whether vanilla decode of layer can run on the CTM kernels.

bool trt_edgellm::canUseXqaDecode(

AttentionLayerTraits const &layer

) noexcept

Whether an XQA decode kernel exists for layer.

CompactPrefillKernel trt_edgellm::selectCompactPrefillKernel(

AttentionLayerTraits const &layer

) noexcept

The ragged paged FMHA kernel the compact prefill rows of layer run on; kNONE when none covers it.

CompactPhaseSupport trt_edgellm::attentionCompactSupport(

AttentionLayerTraits const &layer

) noexcept

AttentionPlugin: phase 8 needs a layer kind the compact path covers, a decode kernel (CTM, or XQA off the CTM-only SMs) and a ragged paged prefill kernel.

std::string trt_edgellm::attentionLayerShape(

AttentionLayerTraits const &layer

)

The heads, head dim and KV-cache traits of layer, as the runtime’s compact capability reason and the plugin’s phase-8 refusal both name it.

CompactPhaseSupport trt_edgellm::causalConv1dCompactSupport(

int32_t smVersion,

int32_t dim,

int32_t width,

int32_t dilation,

int32_t groups,

bool specVerifyState

) noexcept

CausalConv1dPlugin: phase 8 needs a depthwise layer (groups 0 or dim) without spec-verify state and the packed-varlen conv kernel.

CompactPhaseSupport trt_edgellm::gatedDeltaNetCompactSupport(

int32_t kDim,

int32_t vDim,

int32_t smVersion,

bool specVerifyState,

bool diffusionState

) noexcept

GatedDeltaNetPlugin: phase 8 needs no spec-verify or diffusion state and the packed-varlen GDN kernel.

CompactPhaseSupport trt_edgellm::mamba2SsdCompactSupport(

int32_t headDim,

int32_t stateDim,

int32_t smVersion,

bool specVerifyState

) noexcept

MambaPlugin: phase 8 needs no spec-verify state and a packed SSD kernel (the Blackwell datacenter one, or the SM120/121 state-pool one).