Gdn Tree Chunk Kernels#

bool trt_edgellm::kernel::gdnTreeChunkVerifyEnabled(int32_t treeSize)#

Single source of truth for “is the stateless chunk-form verify used for a

tree of this size?” The plugin verify dispatch and the decoder commit path MUST both gate on this: if they disagreed, one side could skip replay while the other expects replay stash, silently corrupting the committed recurrent state. Callers add their own context guards (plugin: DDTree verify phase; decoder: hybrid-state commit).

size_t trt_edgellm::kernel::gdnTreeVerifyRowBytes(
int32_t h,
int32_t hv,
int32_t numNodes
)#

Bytes one request occupies in a layer’s verify stash buffer for a tree of numNodes, rounded up to 256 B: the replay stash, plus the split kernels’ KS/QS and prep scratch when numNodes takes the split path on this device (gdnTreeVerifyFusedFits). Verify writes and the replay commit read rows at this stride.

rt::Coords trt_edgellm::kernel::gdnTreeVerifyStashShape(
bool fp32State,
int32_t convDim,
int32_t maxBatch,
int32_t maxNodes,
int32_t hv,
int32_t dk,
int32_t dv
)#

Shape of a layer’s spec-verify stash for maxBatch rows of maxNodes verify nodes. A layer the chunk-form verify serves (FP32 state; q, k and v heads 128 wide, fused in a conv of convDim channels) needs only rows of gdnTreeVerifyRowBytes, held as [maxBatch, maxNodes, FP32 words], which also fit the rows of smaller trees; any other layer keeps one state per node, [maxBatch, maxNodes, hv, dk, dv]. An active [batch, nodes] keeps the trailing dims.

cudaError_t trt_edgellm::kernel::gdnTreeBuildAncestorMasks(
int32_t const *parentIds,
uint32_t *masksOut,
int32_t batch,
int32_t numNodes,
int32_t maxDepth,
cudaStream_t stream
)#

Build packed inclusive ancestor masks from DDTree parent ids. parentIds [GPU int32]: [batch, numNodes], root/padding = -1 (only node 0 may be a root; padding nodes have parent -1 at index > 0 and mask 0). masksOut [GPU uint32]: [batch, numNodes, kGDN_TREE_CHUNK_MASK_WORDS]. Returns the kernel launch status (cudaSuccess on success); the caller must propagate a failure rather than continue with an unwritten output.

bool trt_edgellm::kernel::gdnTreeVerifyFusedFits(int32_t numNodes)#

True when the fused verify kernel’s shared memory for a numNodes tree fits the current device.

cudaError_t trt_edgellm::kernel::gdnTreeVerifyChunk(
float const *h0,
int32_t const *stateIndices,
int32_t statePoolRows,
__half const *q,
__half const *k,
__half const *v,
__half const *a,
__half const *b,
float const *A_log,
__half const *dt_bias,
int32_t const *parentIds,
uint32_t *masks,
__half *o,
void *stash,
size_t stashBatchStrideBytes,
int32_t batch,
int32_t numNodes,
int32_t h,
int32_t hv,
float scale,
bool useQKL2Norm,
cudaStream_t stream,
GdnTreeVerifyPath path
)#

Chunk-form tree verify. Reads h0 (fp32, READ-ONLY), emits per-node outputs o (fp16) and the replay stash. No state is written anywhere. Shapes follow the GDN plugin contract: q,k [batch, N, h, 128] fp16; v [batch, N, hv, 128] fp16; a,b [batch, N, hv] fp16; A_log [hv] fp32; dt_bias [hv] fp16; h0 [statePoolRows, hv, 128, 128] fp32, the resident state pool: request b reads row stateIndices[b] (row b when stateIndices is nullptr); a row outside [0, statePoolRows) reads as a zero state; o [batch, N, hv, 128] fp16; parentIds [batch, N] int32 DDTree parents (root/padding = -1); masks [batch, N, kGDN_TREE_CHUNK_MASK_WORDS] uint32 scratch for the split path’s ancestor masks; stash: base pointer of this layer’s intermediate buffer; stride stashBatchStrideBytes between batch rows; nodes packed at gdnTreeStashNodeBytes() intervals. On the split path the unused row tail past the last stash cell also holds the verify scratch (KS/QS + prep blocks: align256(N * nodeBytes) + hv*2*ceil16(N)*128 f32 + hv*kGDN_TREE_CHUNK_PREP_WORDS f32); the launcher validates capacity and refuses to launch on overflow. Returns the launch status: cudaSuccess, or the launch/attribute error, or cudaErrorInvalidValue if the scratch does not fit the row stride. The caller MUST propagate a non-success return — continuing would consume garbage verify output and, via the unwritten stash, corrupt the persistent recurrent state on the subsequent replay commit.

cudaError_t trt_edgellm::kernel::gdnTreeReplayCommitBatched(
MtpLayerInfo const *deviceLayerInfos,
int32_t numLayers,
size_t stashBatchStrideBytes,
int32_t const *acceptedIndices,
int32_t const *acceptLengths,
int32_t const *stateIndices,
int32_t batch,
int32_t residentPoolRows,
int32_t maxAcceptLen,
int32_t numNodes,
int32_t h,
int32_t hv,
cudaStream_t stream
)#

Batched replay commit across ALL recurrent layers in one launch (grid = [batch*hv, numLayers]). Per layer, advances the persistent state (MtpLayerInfo::recurrentDst) along the accepted path using the stash the chunk-form verify wrote at the head of MtpLayerInfo::recurrentSrc. Identical math/op-order to a sequential scan over the accepted nodes. acceptedIndices [GPU int32]: [batch, maxAcceptLen] verify-node indices; nullptr selects the chain prefix. acceptLengths [GPU int32]: [batch]. Returns the kernel launch status (cudaSuccess on success); the caller must propagate a failure rather than continue with a partially-committed state.