Embedding Kernels#
-
struct EmbeddingTableDescriptor#
Describes the physical representation of an embedding table.
Keeping the logical shape separate from the packed shape prevents the runtime from interpreting the byte width of an NVFP4 tensor as its hidden size.
Public Members
-
EmbeddingTableFormat format = {EmbeddingTableFormat::kFP16}#
-
trt_edgellm::DataType packedDataType = {trt_edgellm::DataType::kHALF}#
-
int64_t groupSize = {0}#
-
trt_edgellm::DataType scaleDataType = {trt_edgellm::DataType::kFLOAT}#
-
float multiplierGlobalScale = {1.0F}#
-
float modelOutputScale = {1.0F}#
-
EmbeddingTableFormat format = {EmbeddingTableFormat::kFP16}#
- void trt_edgellm::kernel::generateMultimodalIndices(
- rt::Tensor const &inputIds,
- rt::Tensor &multimodalIndices,
- std::optional<int32_t> imageTokenId = std::nullopt,
- std::optional<int32_t> audioTokenId = std::nullopt,
- cudaStream_t stream = nullptr,
- int32_t const *imageBaseOffsets = nullptr,
- int32_t const *audioBaseOffsets = nullptr
Generate per-position multimodal indices on-device from GPU token IDs.
For each position holding an image/audio placeholder token, writes the running count of that modality’s placeholders seen so far (the row of imageEmbeds/audioEmbeds to insert); other positions get -1. Counters are global across the whole [batchSize, seqLen] range in batch-major order, matching the host
generateMultimodalIndicesreference. Runs entirely onstream— no host round-trip, so no D2H/H2D copy is needed to feedembeddingLookup/assembleDeepstackEmbedding.When KV cache prefix reuse is active, the suffix token IDs do not include media tokens from the reused prefix, but the embedding tensor still contains rows for ALL media items (prefix + suffix). The optional per-batch base offsets shift each batch item’s starting counter so suffix tokens index past the prefix rows. Without offsets (nullptr), counters start at 0 as before.
- Parameters:
inputIds – [in] GPU token IDs [batchSize, seqLen] (INT32)
multimodalIndices – [out] GPU indices [batchSize, seqLen] (INT32), same element count as inputIds
imageTokenId – [in] Image placeholder token id, or std::nullopt if no image
audioTokenId – [in] Audio placeholder token id, or std::nullopt if no audio
stream – [in] CUDA stream for execution
imageBaseOffsets – [in] Device pointer to per-batch image index offsets [batchSize], or nullptr
audioBaseOffsets – [in] Device pointer to per-batch audio index offsets [batchSize], or nullptr
- void trt_edgellm::kernel::generateVisionBlockIds( )#
Generate contiguous per-request image-block IDs on the device. Text and padding tokens receive
-1.
- void trt_edgellm::kernel::assembleDeepstackEmbedding(
- rt::Tensor const &inputIds,
- rt::Tensor const &deepstackFeatures,
- rt::Tensor &deepstackEmbeds,
- cudaStream_t stream,
- int32_t imageTokenId = 0,
- rt::OptionalInputTensor multimodalIndices = std::nullopt
Assemble deepstack embeddings by extracting image token embeddings from deepstack features.
This function processes input token IDs and selectively extracts embeddings for image tokens from the provided deepstack features. Image tokens are identified by the explicit imageTokenId, and multimodalIndices selects the deepstack feature row for each image position.
Note
Element offsets are 64-bit: deepstackFeatures and deepstackEmbeds may hold more than 2^32 elements.
- Parameters:
inputIds – [in] Input token IDs with shape [batchSize, seqLen]
deepstackFeatures – [in] Deepstack image features with shape [numImageTokens, hiddenSize]
imageTokenId – [in] Image token ID; positions with this id receive a deepstack feature row
multimodalIndices – [in] Pre-computed indices for image embeddings [batchSize, seqLen]
deepstackEmbeds – [out] Output embeddings with shape [batchSize, seqLen, hiddenSize]
stream – [in] CUDA stream for execution
- Throws:
std::runtime_error – if tensor shapes or data types are invalid
Warning
doxygenfunction: Unable to resolve function “trt_edgellm::kernel::embeddingLookup” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:
- void embeddingLookup(rt::Tensor const &inputIds, rt::Tensor const &embeddingTable, rt::OptionalInputTensor scales, EmbeddingTableDescriptor const &descriptor, rt::Tensor &output, cudaStream_t stream, rt::OptionalInputTensor multimodalIndices = std::nullopt, std::optional<int32_t> imageTokenId = std::nullopt, rt::OptionalInputTensor imageEmbeds = std::nullopt, std::optional<int32_t> audioTokenId = std::nullopt, rt::OptionalInputTensor audioEmbeds = std::nullopt)
- void embeddingLookup(rt::Tensor const &inputIds, rt::Tensor const &embeddingTable, rt::OptionalInputTensor scales, rt::Tensor &output, cudaStream_t stream, rt::OptionalInputTensor multimodalIndices = std::nullopt, std::optional<int32_t> imageTokenId = std::nullopt, rt::OptionalInputTensor imageEmbeds = std::nullopt, std::optional<int32_t> audioTokenId = std::nullopt, rt::OptionalInputTensor audioEmbeds = std::nullopt)