Context Cache Request

class ContextCacheRequest

Owns one admitted runtime request and binds it to the context-cache coordinator lifecycle.

Public Functions

ContextCacheRequest(ContextCacheRequest&&) noexcept = default

ContextCacheRequest &operator=(ContextCacheRequest&&) = delete

ContextCacheRequest(ContextCacheRequest const&) = delete

ContextCacheRequest &operator=(ContextCacheRequest const&) = delete

~ContextCacheRequest() = default

std::vector<int32_t> const &prefillStarts() const noexcept

Per-sequence logical offsets at which runtime prefill begins.

bool preparePrefill()

bool enqueuePrefillCaptures()

bool completePrefill(

DecodingInferenceContext const &context,

std::vector<int32_t> const &commonStateLengths

)

bool prepareDecodeStep(DecodingInferenceContext const &context)

bool completeDecodeStep(

DecodingInferenceContext const &context,

std::vector<int32_t> const &commonStateLengths

)

bool beginBatchCompaction(

std::vector<int32_t> const &oldToNew,

int32_t newBatchSize,

Tensor &deviceBatchMapping

)

bool completeBatchCompaction()

bool finish()

Public Static Functions

static std::optional<ContextCacheRequest> begin(

ContextCacheCoordinator &coordinator,

LLMGenerationRequest const &request,

DecodingInferenceContext const &context,

DecodingStrategyKind strategyKind,

std::vector<int32_t> const &mediaTokenIds = {}

)

Admit one tokenized request and bind its cache resources. A disengaged result means admission failed.

Parameters:

mediaTokenIds – Placeholder token IDs for media modalities (e.g. image, audio). Positions matching any of these IDs are content-hashed for cache differentiation.