Mtp State Scatter Kernels#

struct MtpLayerInfo#

Per-layer pointer bundle for batched MTP state scatter (analogous to KVLayerInfo).

Public Members

void *recurrentDst#

Main recurrent state pool [residentPoolRows, recElements] FP32.

void *recurrentSrc#

Recurrent replay stash, with layout owned by the recurrent backend.

void *convDst#

Main conv state pool [residentPoolRows, convElements] FP16.

void *convSrc#

Intermediate conv state buffer [batchSize, verifyTreeSize, convElements] FP16.

void trt_edgellm::kernel::mtpScatterConvStates(
MtpLayerInfo const *deviceLayerInfos,
int32_t numLayers,
int32_t activeBatchSize,
int32_t residentPoolRows,
int32_t verifyTreeSize,
int32_t stateElements,
int32_t const *acceptLengths,
int32_t const *stateIndices,
cudaStream_t stream
)#

Commit each accepted conv prefix endpoint to its selected resident row. Zero-length prefixes preserve the resident state. stateElements must be divisible by 8.

void trt_edgellm::kernel::mtpScatterAcceptedTreeConvStates(
MtpLayerInfo const *deviceLayerInfos,
int32_t numLayers,
int32_t activeBatchSize,
int32_t residentPoolRows,
int32_t verifyTreeSize,
int32_t stateElements,
int32_t const *acceptedStateNodeIds,
int32_t maxAcceptLen,
int32_t const *acceptLengths,
int32_t const *stateIndices,
cudaStream_t stream
)#

Commit the last non-negative node within each accepted path to its selected resident conv row. Paths without an accepted node preserve the resident state. stateElements must be divisible by 8.