Context Cache Request#
-
class ContextCacheRequest#
Owns the cache-side working set of one batch – the sequences a blocking request brought with it, or the ones a long-lived session admits one at a time – and binds it to the context-cache coordinator lifecycle.
Sequence-admission transaction (see ContextCacheCoordinator). Several sequences may be in
admission at once; each prefill chunk runs prepare -> (final chunk: capture) -> stream sync -> complete -> advance (non-final) or finalize (final).
-
bool prepareSequenceAdmission(int32_t sequenceIndex)#
Open the sequence’s next prefill chunk; the first one restores any hybrid checkpoint into its resident slot.
-
bool enqueueSequenceAdmissionCapture(int32_t sequenceIndex)#
Enqueue the sequence’s prompt-end hybrid checkpoint capture after its final chunk, before the stream sync.
-
bool completeSequenceAdmissionDeviceWork(int32_t sequenceIndex)#
Record that the runtime’s stream synchronization completed the sequence’s open chunk.
- bool advanceSequenceAdmission(
- int32_t sequenceIndex,
- int32_t committedLength
Record a committed non-final chunk: the slot holds the first
committedLengthprompt tokens. Nothing is published.
-
bool abortSequenceAdmission(int32_t sequenceIndex)#
Discard a failed or cancelled sequence-admission transaction before evicting its slot; nothing of the sequence is published.
- bool finalizeSequenceAdmission(
- int32_t sequenceIndex,
- int32_t const &lookaheadToken,
- int32_t fullInputLength,
- std::optional<int32_t> commonStateLength = std::nullopt
Record the final chunk’s lookahead token so the slot’s ledger matches its cached state, then publish the ready prefix. Call once, right after the final chunk succeeds.
commonStateLengthis the prefix a speculative sequence’s draft has seated (see ContextCacheCoordinator::finalizeSequenceAdmission).
Public Types
-
enum class AdmitSequenceStatus : uint8_t#
Values:
-
enumerator kAdmitted#
-
enumerator kNoCapacity#
Transient pool pressure: pages free as resident sequences retire, so the caller may retry at a later step boundary rather than failing the request.
-
enumerator kOverlongPrefill#
The uncached suffix exceeds the engine input limit. Nothing was leased or evicted.
-
enumerator kFailed#
-
enumerator kAdmitted#
Public Functions
-
ContextCacheRequest(ContextCacheRequest&&) noexcept = default#
-
ContextCacheRequest &operator=(ContextCacheRequest&&) = delete#
-
ContextCacheRequest(ContextCacheRequest const&) = delete#
-
ContextCacheRequest &operator=(ContextCacheRequest const&) = delete#
-
~ContextCacheRequest() = default#
-
std::vector<int32_t> const &prefillStarts() const noexcept#
Per-sequence logical offsets at which runtime prefill begins.
-
int32_t reuseTokenLength(int32_t slot) const noexcept#
Number of reused prefix tokens for a slot (== the logical prefill start offset).
-
std::vector<SwaKVCacheState> takeSwaStates()#
-
bool isExternallyOwnedSwaPage(int32_t slot, PageId page) const#
-
PageId replaceSwaPage(int32_t slot, PageId page)#
- AdmitSequenceStatus admitSequence(
- std::vector<int32_t> const &tokenIds,
- std::string const &loraWeightsName,
- DecodingKvHeadroom const &headroom,
- int32_t &prefillStart,
- ResidentRef resident,
- cudaStream_t stream,
- ContextCacheLookupPolicy lookupPolicy,
- ContextCacheCommitPolicy commitPolicy,
- std::vector<int32_t> const &mediaTokenIds = {},
- std::vector<Hash128> const &imageContentDigests = {},
- std::vector<audioUtils::AudioData> const &audioBuffers = {}
Join one more sequence to this live request: lookup, lease, and row binding. On kAdmitted,
prefillStartreceives the reused prefix length the seated prefill skips; on kOverlongPrefill, the reused prefix length the lookup found.imageContentDigestsholds one identity per image of the sequence, in prompt order; its media positions key on them the same way begin() keys the first sequences’.lookupPolicyandcommitPolicyare the sequence’s own; the caller has already applied whatever forces bypass (see lookupPolicyFor).
-
bool retractSequenceAdmission() noexcept#
Undo the most recent admitSequence before its slot ever joined the runtime batch: the recovery path for a seating that threw between lease and slot append.
- bool publishHybridSpecEndpoint(
- int32_t slot,
- int32_t residentStateLength,
- Tensor const &baseHiddenStates,
- int32_t boundaryHiddenRow
Publish one Hybrid+spec checkpoint at the stable predecessor boundary. Forwards to the coordinator’s dedicated MTP publication entrypoint; the runtime drives this after the folded draft prefill materialized boundary state.
- bool restoreHybridSpecBoundaryHidden(
- int32_t slot,
- Tensor &baseHiddenStates,
- int32_t destinationRow
Restore the reused checkpoint’s saved boundary base-hidden row into baseHiddenStates for the fold micro-forward.
-
bool preparePrefill()#
-
bool enqueuePrefillCaptures()#
- bool completePrefill(
- DecodingInferenceContext const &context,
- std::vector<int32_t> const &commonStateLengths,
- SwaWindowStateView const *swaWindowStates = nullptr
- bool prepareDecodeStep(
- DecodingInferenceContext const &context,
- DecodingKvHeadroom const &headroom,
- std::vector<int32_t> const *extraTokensPerSlot = nullptr,
- std::vector<int32_t> const *draftExtraTokensPerSlot = nullptr
extraTokensPerSlot/draftExtraTokensPerSlot:each row’s own base and draft growth when jump-forward changes it (see the coordinator).
- bool prepareDecodeStep(
- DecodingInferenceContext const &context,
- DecodingKvHeadroom const &headroom,
- std::vector<int32_t> const &decodeRows
Prepare a step in which only
decodeRows(execution rows with a complete prompt) decode. The matching completeDecodeStep gives every other row the hold advance and publishes none of them, whatever the row appended this step (a final prefill chunk’s first token) or not (a paused row, a non-final chunk).
- bool completeDecodeStep(
- DecodingInferenceContext const &context,
- std::vector<int32_t> const &commonStateLengths,
- SwaWindowStateView const *swaWindowStates = nullptr
- bool beginBatchCompaction(
- std::vector<int32_t> const &oldToNew,
- int32_t newBatchSize,
- Tensor &deviceBatchMapping
- bool completeBatchCompaction(
- std::vector<int32_t> const &keepMapping
Compact the per-slot reuse bookkeeping to survivors.
keepMappingis the runtime’s oldSlot -> newSlot batch mapping (-1 for an evicted slot), the same one performBatchEvict uses; without it the reuse-length vector desyncs from the batch after the first eviction.
-
bool finish()#
Public Static Functions
- static std::optional<ContextCacheRequest> begin(
- ContextCacheCoordinator &coordinator,
- ContextCacheBatchAdmission const &admission,
- DecodingKvHeadroom const &headroom,
- cudaStream_t stream
Open a batch. The admission names what the deployment fixes for every sequence the batch will hold, and the sequences it starts with, if any; a session starts with none and takes each one through admitSequence(). The result is left admitted; preparePrefill() enters execution. A disengaged result means admission failed. Throws the EDGELLM_INPUT_TOO_LONG error when a sequence’s uncached suffix exceeds the engine input limit; nothing is leased or evicted for that sequence.
- static std::optional<ContextCacheRequest> begin(
- ContextCacheCoordinator &coordinator,
- LLMGenerationRequest const &request,
- DecodingInferenceContext const &context,
- bool speculativeRequest,
- DecodingKvHeadroom const &headroom,
- std::vector<int32_t> const &mediaTokenIds = {},
- DecodingTokenStateContract tokenStateContract = DecodingTokenStateContract::kCommittedPlusLookahead,
- ContextCacheCommitPolicy commitPolicy = ContextCacheCommitPolicy::kIncludingGeneratedTokens
Open a batch holding one tokenized request’s sequences, under the request’s cache policies. A disengaged result means admission failed.
- Parameters:
mediaTokenIds – Placeholder token IDs for media modalities (e.g. image, audio). Positions matching any of these IDs key on the media’s content identity: an image’s from DecodingInferenceContext::imageContentDigests, an audio clip’s from its PCM bytes, or its mel-spectrogram path when it carries none. A request with an image that has no identity bypasses the cache.
- static ContextCacheLookupPolicy lookupPolicyFor(
- LLMGenerationRequest const &request,
- bool outputThinkerEmbeddings,
- std::vector<int32_t> const &mediaTokenIds,
- std::vector<std::vector<Hash128>> const &imageDigests
The lookup policy a request runs under: its own, unless something forces bypass – speech output, thinker hidden-state capture, or an image without a content identity (which would key on the media token alone, every image’s placeholder).
mediaTokenIdsis the deployment’s media placeholder list; empty means the deployment hashes no media.imageDigestsholds one identity per image of each sequence, as multimodal preprocessing computed them.
- static ContextCacheLookupPolicy sequenceLookupPolicy(
- ContextCacheLookupPolicy requestPolicy,
- std::optional<ContextCacheLookupPolicy> sequencePolicy
The policy one sequence of a request runs under: the request’s (lookupPolicyFor), which the sequence’s own
sequencePolicymay narrow to bypass but never widen – a request-wide bypass binds every sequence.
-
bool prepareSequenceAdmission(int32_t sequenceIndex)#
-
struct MediaPlaceholderRun#
One maximal span of a media placeholder token in a prompt: one media item, in prompt order. A transition between placeholder tokens (an image run directly followed by an audio run) closes the run, as the context cache’s per-position media hash keys it.
-
struct MediaPlaceholderCounts#
How many media items a prompt’s placeholders stand for, per modality: the image token is mediaTokenIds().front(), every other placeholder is audio.
- std::vector<MediaPlaceholderRun> trt_edgellm::rt::mediaPlaceholderRuns(
- std::vector<int32_t> const &tokenIds,
- std::vector<int32_t> const &mediaTokenIds
The media runs of
tokenIds, for the deployment’s placeholder tokens (LLMRankRuntime::mediaTokenIds()).
- MediaPlaceholderCounts trt_edgellm::rt::countMediaPlaceholders(
- std::vector<int32_t> const &tokenIds,
- std::vector<int32_t> const &mediaTokenIds
- int32_t trt_edgellm::rt::deriveContextCacheReplayTailLength(
- chat_template::ChatTemplate const &chatTemplate,
- tokenizer::Tokenizer const &tokenizer,
- LLMGenerationRequest const &request,
- std::vector<int32_t> const &promptTokenIds
Derive the unstable generation-prompt suffix for Hybrid+spec context reuse: the tokens of
promptTokenIdsafter the prefix a committed assistant turn renders the same way (plus the one successor token MTP needs).This deliberately renders a committed assistant probe in addition to the already prepared generation prompt. Comparing those token streams identifies the stable prefix that can remain cached. Used only when the caller requests the -1 automatic-derivation sentinel.