Gemma4 Embedding Preprocessor#
-
class Gemma4EmbeddingPreprocessor#
Runtime preprocessor for Gemma4 E-model per-layer embeddings (PLE).
Owns a resident/cached pool and a token-to-slot map shared by all PLE layers. The supplied stream outlives the preprocessor and is used by all engine consumers.
Public Functions
- LLMEngineConfig const &config,
- int32_t maxBatchSize,
- int32_t maxSeqLen,
- TensorMap &tensorMap,
- cudaStream_t stream,
- PleRuntimeConfig const &pleConfig = {},
- std::filesystem::path const &checkpointDir = {},
- std::shared_ptr<PleIoBudget> pleIoBudget = nullptr
-
~Gemma4EmbeddingPreprocessor() noexcept#
- void embed(
- Tensor const &tokenIds,
- cudaStream_t stream,
- std::vector<int32_t> const &queryLengths = {},
- Tensor const *deviceQueryLengths = nullptr,
- Tensor const *hostTokenIds = nullptr
Gather PLE tensors for the current token-id tensor shape. Device query lengths mask entry padding and must remain valid through queued work. Cached residency requires matching host lengths and a CPU INT32 token snapshot in physical row order. The host snapshot is read synchronously; resident mode can use GPU-only inputs.
- void embedPacked( )#
Gather PLE tensors for a packed token-major step whose every token ID is a real token (no row padding). The token count may exceed one request’s maximum sequence length, up to the allocated map capacity. Cached residency requires a CPU INT32 snapshot of the same token IDs.
-
void reshapeOutputsTokenMajor(int64_t physicalTokens)#
Reshape already-bound token-major outputs for an execution or CUDA-graph capture shape.
-
bool needsHostTokens() const noexcept#
Whether embed needs host token IDs, which only cached residency does.
-
inline int64_t distinctRowCapacity() const noexcept#
The most distinct table rows one gather can map: every table row under resident storage, the cache slots less the reserved zero slot under cached storage.
-
struct PooledResources#
Public Functions
- PleTable table,
- int32_t slots,
- cudaStream_t stream,
- int32_t maxBatch,
- int32_t maxSeq,
- PleResidency mode,
- std::shared_ptr<PleIoBudget> pleIoBudget
Public Members
-
PleRowStore store#
-
PleResidency residency#
-
std::vector<PleRowStore::RequestLease> leases#