Embedding Kernels#

struct EmbeddingTableDescriptor#

Describes the physical representation of an embedding table.

Keeping the logical shape separate from the packed shape prevents the runtime from interpreting the byte width of an NVFP4 tensor as its hidden size.

Public Members

EmbeddingTableFormat format = {EmbeddingTableFormat::kFP16}#
rt::Coords logicalShape#
rt::Coords packedShape#
trt_edgellm::DataType packedDataType = {trt_edgellm::DataType::kHALF}#
int64_t groupSize = {0}#
rt::Coords scaleShape#
trt_edgellm::DataType scaleDataType = {trt_edgellm::DataType::kFLOAT}#
float multiplierGlobalScale = {1.0F}#
float modelOutputScale = {1.0F}#
void trt_edgellm::kernel::generateMultimodalIndices(
rt::Tensor const &inputIds,
rt::Tensor &multimodalIndices,
std::optional<int32_t> imageTokenId = std::nullopt,
std::optional<int32_t> audioTokenId = std::nullopt,
cudaStream_t stream = nullptr,
int32_t const *imageBaseOffsets = nullptr,
int32_t const *audioBaseOffsets = nullptr
)#

Generate per-position multimodal indices on-device from GPU token IDs.

For each position holding an image/audio placeholder token, writes the running count of that modality’s placeholders seen so far (the row of imageEmbeds/audioEmbeds to insert); other positions get -1. Counters are global across the whole [batchSize, seqLen] range in batch-major order, matching the host generateMultimodalIndices reference. Runs entirely on stream — no host round-trip, so no D2H/H2D copy is needed to feed embeddingLookup/assembleDeepstackEmbedding.

When KV cache prefix reuse is active, the suffix token IDs do not include media tokens from the reused prefix, but the embedding tensor still contains rows for ALL media items (prefix + suffix). The optional per-batch base offsets shift each batch item’s starting counter so suffix tokens index past the prefix rows. Without offsets (nullptr), counters start at 0 as before.

Parameters:
  • inputIds – [in] GPU token IDs [batchSize, seqLen] (INT32)

  • multimodalIndices – [out] GPU indices [batchSize, seqLen] (INT32), same element count as inputIds

  • imageTokenId – [in] Image placeholder token id, or std::nullopt if no image

  • audioTokenId – [in] Audio placeholder token id, or std::nullopt if no audio

  • stream – [in] CUDA stream for execution

  • imageBaseOffsets – [in] Device pointer to per-batch image index offsets [batchSize], or nullptr

  • audioBaseOffsets – [in] Device pointer to per-batch audio index offsets [batchSize], or nullptr

void trt_edgellm::kernel::generateVisionBlockIds(
rt::Tensor const &inputIds,
rt::Tensor &visionBlockIds,
int32_t imageTokenId,
cudaStream_t stream
)#

Generate contiguous per-request image-block IDs on the device. Text and padding tokens receive -1.

void trt_edgellm::kernel::assembleDeepstackEmbedding(
rt::Tensor const &inputIds,
rt::Tensor const &deepstackFeatures,
rt::Tensor &deepstackEmbeds,
cudaStream_t stream,
int32_t imageTokenId = 0,
rt::OptionalInputTensor multimodalIndices = std::nullopt
)#

Assemble deepstack embeddings by extracting image token embeddings from deepstack features.

This function processes input token IDs and selectively extracts embeddings for image tokens from the provided deepstack features. Image tokens are identified by the explicit imageTokenId, and multimodalIndices selects the deepstack feature row for each image position.

Note

Element offsets are 64-bit: deepstackFeatures and deepstackEmbeds may hold more than 2^32 elements.

Parameters:
  • inputIds – [in] Input token IDs with shape [batchSize, seqLen]

  • deepstackFeatures – [in] Deepstack image features with shape [numImageTokens, hiddenSize]

  • imageTokenId – [in] Image token ID; positions with this id receive a deepstack feature row

  • multimodalIndices – [in] Pre-computed indices for image embeddings [batchSize, seqLen]

  • deepstackEmbeds – [out] Output embeddings with shape [batchSize, seqLen, hiddenSize]

  • stream – [in] CUDA stream for execution

Throws:

std::runtime_error – if tensor shapes or data types are invalid

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::kernel::embeddingLookup” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:

- void embeddingLookup(rt::Tensor const &inputIds, rt::Tensor const &embeddingTable, rt::OptionalInputTensor scales, EmbeddingTableDescriptor const &descriptor, rt::Tensor &output, cudaStream_t stream, rt::OptionalInputTensor multimodalIndices = std::nullopt, std::optional<int32_t> imageTokenId = std::nullopt, rt::OptionalInputTensor imageEmbeds = std::nullopt, std::optional<int32_t> audioTokenId = std::nullopt, rt::OptionalInputTensor audioEmbeds = std::nullopt)
- void embeddingLookup(rt::Tensor const &inputIds, rt::Tensor const &embeddingTable, rt::OptionalInputTensor scales, rt::Tensor &output, cudaStream_t stream, rt::OptionalInputTensor multimodalIndices = std::nullopt, std::optional<int32_t> imageTokenId = std::nullopt, rt::OptionalInputTensor imageEmbeds = std::nullopt, std::optional<int32_t> audioTokenId = std::nullopt, rt::OptionalInputTensor audioEmbeds = std::nullopt)