Managed KV Cache Request#

class ManagedKVCacheRequest#

One runtime request facade over orthogonal execution paging and context-reuse clients.

Context reuse owns cache-record references while bounded SWA owns sparse-table transitions. Either client may be active alone, or both may be composed for vanilla bounded-SWA context reuse.

In-flight admission, context-reuse backend only.

The bounded-SWA backend has no live-slot admission (its per-slot window state cannot join a running batch), and supportsSeatedAdmission() refuses such deployments before an engine that would call these is ever constructed; the kFailed/false returns here are the backstop.

ContextCacheRequest::AdmitSequenceStatus admitSequence(
std::vector<int32_t> const &tokenIds,
std::string const &loraWeightsName,
DecodingKvHeadroom const &headroom,
int32_t &prefillStart,
ResidentRef resident,
cudaStream_t stream,
ContextCacheLookupPolicy lookupPolicy,
ContextCacheCommitPolicy commitPolicy,
std::vector<int32_t> const &mediaTokenIds = {},
std::vector<Hash128> const &imageContentDigests = {},
std::vector<audioUtils::AudioData> const &audioBuffers = {}
)#
bool prepareSequenceAdmission(int32_t sequenceIndex)#
bool enqueueSequenceAdmissionCapture(int32_t sequenceIndex)#
bool completeSequenceAdmissionDeviceWork(int32_t sequenceIndex)#
bool advanceSequenceAdmission(
int32_t sequenceIndex,
int32_t committedLength
)#
bool abortSequenceAdmission(int32_t sequenceIndex)#
bool finalizeSequenceAdmission(
int32_t sequenceIndex,
int32_t const &lookaheadToken,
int32_t fullInputLength,
std::optional<int32_t> commonStateLength = std::nullopt
)#
bool retractSequenceAdmission() noexcept#

Public Functions

ManagedKVCacheRequest(ManagedKVCacheRequest&&) noexcept = default#
ManagedKVCacheRequest &operator=(ManagedKVCacheRequest&&) = delete#
ManagedKVCacheRequest(ManagedKVCacheRequest const&) = delete#
ManagedKVCacheRequest &operator=(
ManagedKVCacheRequest const&
) = delete#
~ManagedKVCacheRequest() = default#
bool hasContextReuse() const noexcept#
bool hasBoundedSwa() const noexcept#
std::vector<int32_t> const *prefillStarts() const noexcept#
int32_t reuseTokenLength(int32_t slot) const#
bool publishHybridSpecEndpoint(
int32_t slot,
int32_t residentStateLength,
Tensor const &baseHiddenStates,
int32_t boundaryHiddenRow
)#
bool restoreHybridSpecBoundaryHidden(
int32_t slot,
Tensor &baseHiddenStates,
int32_t destinationRow
)#
bool preparePrefill()#
bool prepareSwaPrefill(
std::vector<int32_t> const &effectiveInputLengths
)#
bool enqueuePrefillCaptures()#
bool completePrefill(
DecodingInferenceContext const &context,
std::vector<int32_t> const &commonStateLengths
)#
bool prepareDecodeStep(
DecodingInferenceContext const &context,
DecodingKvHeadroom const &headroom,
DecodingKvHeadroom const &jumpForwardHeadroom
)#

Grow every slot for the next decode step. With jump-forward, a vanilla slot grows its pages and its SWA window by its own pending count instead of headroom.baseExtraTokens, and a speculative slot that asked for jump-forward grows by jumpForwardHeadroom instead of headroom.

bool prepareDecodeStep(
DecodingInferenceContext const &context,
DecodingKvHeadroom const &headroom,
std::vector<int32_t> const &decodeRows
)#

Decode-subset form (see ContextCacheRequest::prepareDecodeStep); context-reuse backend only.

bool completeDecodeStep(
DecodingInferenceContext const &context,
std::vector<int32_t> const &commonStateLengths
)#
bool beginBatchCompaction(
std::vector<int32_t> const &oldToNew,
int32_t newBatchSize,
Tensor &deviceBatchMapping
)#
bool completeBatchCompaction(
std::vector<int32_t> const &keepMapping
)#
bool finish()#

Public Static Functions

static std::optional<ManagedKVCacheRequest> begin(
ContextCacheCoordinator *contextCache,
BoundedSwaKVPageManager *swaPageManager,
LLMGenerationRequest const &request,
DecodingInferenceContext const &context,
bool speculativeRequest,
DecodingKvHeadroom const &headroom,
DecodingTokenStateContract tokenStateContract,
ContextCacheCommitPolicy commitPolicy,
std::vector<int32_t> const &mediaTokenIds = {}
)#
static std::optional<ManagedKVCacheRequest> begin(
ContextCacheCoordinator &contextCache,
ContextCacheBatchAdmission const &admission,
DecodingKvHeadroom const &headroom,
cudaStream_t stream
)#

Open a context-reuse batch from a batch admission (see ContextCacheRequest::begin): the form a session uses, starting with no sequence and admitting each one live. Bounded SWA has no live-slot admission, so a batch opened this way carries none; a session on such a deployment is refused before this is reached.