Context Cache Request

class ContextCacheRequest

Owns one admitted runtime request and binds it to the context-cache coordinator lifecycle.

Public Functions

ContextCacheRequest(ContextCacheRequest&&) noexcept = default

ContextCacheRequest &operator=(ContextCacheRequest&&) = delete

ContextCacheRequest(ContextCacheRequest const&) = delete

ContextCacheRequest &operator=(ContextCacheRequest const&) = delete

~ContextCacheRequest() = default

std::vector<int32_t> const &prefillStarts() const noexcept

Per-sequence logical offsets at which runtime prefill begins.

int32_t reuseTokenLength(int32_t slot) const noexcept

Number of reused prefix tokens for a slot (== the logical prefill start offset).

bool publishHybridMtpEndpoint(

int32_t slot,

int32_t residentStateLength,

Tensor const &baseHiddenStates,

int32_t boundaryHiddenRow

)

Publish one Hybrid+MTP checkpoint at the stable predecessor boundary. Forwards to the coordinator’s dedicated MTP publication entrypoint; the runtime drives this after the folded draft prefill materialized boundary state.

bool restoreHybridMtpBoundaryHidden(

int32_t slot,

Tensor &baseHiddenStates,

int32_t destinationRow

)

Restore the reused checkpoint’s saved boundary base-hidden row into baseHiddenStates for the fold micro-forward.

bool preparePrefill()

bool enqueuePrefillCaptures()

bool completePrefill(

DecodingInferenceContext const &context,

std::vector<int32_t> const &commonStateLengths

)

bool prepareDecodeStep(

DecodingInferenceContext const &context,

DecodingKvHeadroom const &headroom

)

bool completeDecodeStep(

DecodingInferenceContext const &context,

std::vector<int32_t> const &commonStateLengths

)

bool beginBatchCompaction(

std::vector<int32_t> const &oldToNew,

int32_t newBatchSize,

Tensor &deviceBatchMapping

)

bool completeBatchCompaction()

bool finish()

Public Static Functions

static std::optional<ContextCacheRequest> begin(

ContextCacheCoordinator &coordinator,

LLMGenerationRequest const &request,

DecodingInferenceContext const &context,

bool speculativeRequest,

DecodingKvHeadroom const &headroom,

std::vector<int32_t> const &mediaTokenIds = {}

)

Admit one tokenized request and bind its cache resources. A disengaged result means admission failed.

Parameters:

mediaTokenIds – Placeholder token IDs for media modalities (e.g. image, audio). Positions matching any of these IDs are content-hashed for cache differentiation.