Gemma4 Embedding Preprocessor#

class Gemma4EmbeddingPreprocessor#

Runtime preprocessor for Gemma4 E-model per-layer embeddings (PLE).

Owns a resident/cached pool and a token-to-slot map shared by all PLE layers. The supplied stream outlives the preprocessor and is used by all engine consumers.

Public Functions

Gemma4EmbeddingPreprocessor(
LLMEngineConfig const &config,
int32_t maxBatchSize,
int32_t maxSeqLen,
TensorMap &tensorMap,
cudaStream_t stream,
PleRuntimeConfig const &pleConfig = {},
std::filesystem::path const &checkpointDir = {},
std::shared_ptr<PleIoBudget> pleIoBudget = nullptr
)#
~Gemma4EmbeddingPreprocessor() noexcept#
void embed(
Tensor const &tokenIds,
cudaStream_t stream,
std::vector<int32_t> const &queryLengths = {},
Tensor const *deviceQueryLengths = nullptr,
Tensor const *hostTokenIds = nullptr
)#

Gather PLE tensors for the current token-id tensor shape. Device query lengths mask entry padding and must remain valid through queued work. Cached residency requires matching host lengths and a CPU INT32 token snapshot in physical row order. The host snapshot is read synchronously; resident mode can use GPU-only inputs.

void embedPacked(
Tensor const &tokenIds,
cudaStream_t stream,
Tensor const *hostTokenIds
)#

Gather PLE tensors for a packed token-major step whose every token ID is a real token (no row padding). The token count may exceed one request’s maximum sequence length, up to the allocated map capacity. Cached residency requires a CPU INT32 snapshot of the same token IDs.

void reshapeOutputsTokenMajor(int64_t physicalTokens)#

Reshape already-bound token-major outputs for an execution or CUDA-graph capture shape.

bool needsHostTokens() const noexcept#

Whether embed needs host token IDs, which only cached residency does.

inline int64_t distinctRowCapacity() const noexcept#

The most distinct table rows one gather can map: every table row under resident storage, the cache slots less the reserved zero slot under cached storage.

struct PooledResources#

Public Functions

inline PooledResources(
PleTable table,
int32_t slots,
cudaStream_t stream,
int32_t maxBatch,
int32_t maxSeq,
PleResidency mode,
std::shared_ptr<PleIoBudget> pleIoBudget
)#

Public Members

PleRowStore store#
Tensor rowMap#
Tensor hostMap#
PleResidency residency#
std::vector<PleRowStore::RequestLease> leases#