Qwen4 Exp Ngram Preprocessor#

void trt_edgellm::rt::advanceHistory(
int32_t residentSlot,
int32_t const *tokens,
int32_t count
)#

Restore one slot’s token history from the tokens already consumed for its request. ! ! The hash reaches back ngramContextLength tokens, so a chunk that does not start a ! sequence has to know what preceded it. That history is not separate state – it is the tail ! of the tokens already consumed – so re-deriving it here keeps a reused prefix and a ! chunked prefill correct without a checkpoint of its own. ! !

Runtime preprocessor for the Qwen4-Exp hashed n-gram embedding (PLE).

Hashes each token’s n-grams into table row IDs and turns them into a row map into the shared embedding row pool. Row IDs depend on token IDs alone, so none of this needs a hidden state and the whole mapping can happen before the engine runs; the lookup itself is a graph operation.

The table is the largest tensor in the model – 47.7 GiB in the released checkpoint – while a forward reads ngramHeads rows per token out of it. It therefore stays in the checkpoint and ! reaches the device through PleRowStore, either resident or as a row cache over SSD. ! ! Unrelated to Gemma4EmbeddingPreprocessor despite the shared “PLE” name. class Qwen4ExpNgramPreprocessor { public: Qwen4ExpNgramPreprocessor(LLMEngineConfig const& config, int32_t maxBatchSize, int32_t maxSeqLen, TensorMap& tensorMap, cudaStream_t stream, PleRuntimeConfig const& pleConfig = {}, std::filesystem::path const& checkpointDir = {}, std::shared_ptr<PleIoBudget> pleIoBudget = nullptr); ~Qwen4ExpNgramPreprocessor() noexcept;

!

! Start one slot’s token history from scratch, as the beginning of a document. !

! Hash the step’s tokens and upload the resulting row map. ! ! The entry-padded token layout is what lets one contiguous map serve the token-major ! binding: every sequence occupies queryWidth rows, so [numSequences, queryWidth, heads] ! and [numSequences * queryWidth, heads]

are the same bytes. ! !

! Reshape the bound row map for an execution or CUDA-graph capture shape. void reshapeOutput(int64_t physicalTokens);

! The row map bound for the current step. ! ! Everything that produces it runs on the host and is bit-reproducible, unlike the engine’s ! own output past the first MoE. That makes it the one probe that can answer “did chunking or

! padding change anything” by exact equality rather than by tolerance. ! !

! Counters for the most recent step; meaningful in cached residency only. PleRowStore::Stats const& stats() const;

private: struct PooledResources; bool drainPooled() noexcept;

! Drop the previous step’s row pins, once the device has finished reading them. void releaseLeases();

! This resident slot’s ngramContextLength preceding tokens. int32_t* historyForSlot(int32_t residentSlot);

! Fold count newly consumed tokens into one resident slot’s history.

Parameters:

  • residentSlot – [in] Resident state-pool row of the sequence, as in state_indices !

  • consumedTokens – [in] Tokens of this request that precede the current call, may be null ! when consumedCount is 0 !

  • consumedCount – [in] Number of such tokens; fewer than the context length pads with EOS, ! which is also what marks the start of a sequence void seedHistory(int32_t residentSlot, int32_t const* consumedTokens, int32_t consumedCount);

  • residentSlot – [in] Resident state-pool row of the sequence void resetHistory(int32_t residentSlot);

  • hostTokenIds – [in] [numSequences, queryWidth] INT32 token IDs in pinned host memory, ! right padded per sequence !

  • hostQueryLengths – [in] [numSequences] INT32 valid token count per sequence, or null ! when every sequence contributes all queryWidth tokens !

  • residentSlots – [in] [numSequences] INT32 state-pool row per sequence, in execution ! order; null means the identity mapping !

  • numSequences – [in] Number of active sequences !

  • queryWidth – [in] Padded token count per sequence !

  • stream – [in] CUDA stream for the upload !

  • hostTreeParents – [in] [numSequences, queryWidth] INT32 parent node per token (-1 for the ! root), or null for a chain. A tree hashes each node over its ! ancestors and leaves the history unchanged. void embed(int32_t const* hostTokenIds, int32_t const* hostQueryLengths, int32_t const* residentSlots, int32_t numSequences, int32_t queryWidth, cudaStream_t stream, int32_t const* hostTreeParents = nullptr);

Returns:

The bound [physicalTokens, ngramHeads] INT32 tensor Tensor const& rowMap() const;

On this page