Context Cache Manager

class CacheRequestLease

Move-only ownership of one request’s active context-cache references and page bindings.

ContextCacheManager provides no internal synchronization. Callers must serialize all manager and lease access under one single-writer execution contract. A lease must be released or destroyed before its manager is destroyed. Destruction releases every active reference; publication separately adds cache ownership for reusable state. Moving a lease transfers its active-reference cleanup responsibility.

Public Functions

CacheRequestLease() noexcept = default

~CacheRequestLease() noexcept

CacheRequestLease(CacheRequestLease &&other) noexcept

CacheRequestLease &operator=(CacheRequestLease &&other) noexcept

CacheRequestLease(CacheRequestLease const&) = delete

CacheRequestLease &operator=(CacheRequestLease const&) = delete

std::vector<PageId> const &basePages() const noexcept

Ordered host binding list of base-model logical K-page IDs; the adapter expands it to a kernel [K, V] row.

std::vector<PageId> const &draftPages() const noexcept

Ordered host binding list of EAGLE draft logical K-page IDs; empty for a vanilla lease.

std::optional<int32_t> recurrentSnapshotSlot() const noexcept

std::optional<int32_t> partialKvSnapshotSlot() const noexcept

bool valid() const noexcept

void release() noexcept

class ContextCacheManager

Host orchestrator for the complete context-cache lifecycle under an externally serialized single-writer contract.

The manager owns neither a worker thread nor a task queue and is not thread-safe. All manager and lease access that could overlap a mutation must be externally serialized, and no lease may outlive its manager. publish() is the final host commit for a producer whose state is already ready; it neither inspects nor waits on CUDA events. A publication must contain at least one resident full block.

The lifecycle is acquire -> model execution -> publish or release. Each acquire method plans and pins the selected hit as one serialized manager operation before eviction and request-private allocation. Publication atomically adds cache ownership, canonical base-block mappings, and one complete record. ResourcePools owns capacity/refcounts, BaseBlockIndex owns canonical base lookup, DraftPathIndex owns coherent EAGLE-path lookup, CacheRecordStore owns endpoints/LRU, and EvictionPlanner selects record victims without mutation.

Public Functions

ContextCacheManager(

int32_t pageSize,

ResourceDemand capacities,

int32_t maxRecords

)

AcquireResult acquireVanilla(

std::vector<BlockHash> const &inputFullBlockHashes,

int32_t inputTokenCount,

ContextCacheLookupPolicy lookupPolicy = ContextCacheLookupPolicy::kUseCache

)

AcquireResult acquireHybrid(

std::vector<HybridCheckpointCandidate> const &candidates,

std::vector<BlockHash> const &inputFullBlockHashes,

int32_t inputTokenCount,

bool hasAttention,

ContextCacheLookupPolicy lookupPolicy = ContextCacheLookupPolicy::kUseCache

)

std::vector<int32_t> hybridCandidateLengths(

int32_t inputTokenCount

) const

AcquireResult acquireSpec(

std::vector<BlockHash> const &inputFullBlockHashes,

int32_t inputTokenCount,

ContextCacheLookupPolicy lookupPolicy = ContextCacheLookupPolicy::kUseCache

)

The caller must first select greedy, non-hybrid EAGLE on a supported full-attention or full-allocation SWA deployment; the manager cannot infer sampling policy or model topology.

bool growBasePages(CacheRequestLease &lease, int32_t count)

Atomically append private base pages; false leaves the lease and cache metadata unchanged.

bool growSpecPages(

CacheRequestLease &lease,

int32_t baseCount,

int32_t draftCount

)

Atomically append EAGLE base and draft pages; false leaves the lease and cache metadata unchanged.

std::optional<HybridSnapshotReservation> reserveHybridSnapshots(

CacheRequestLease &lease,

bool needsPartialKvSnapshot

)

Reserve unpublished hybrid snapshot storage. Failure is retention pressure and leaves the lease unchanged.

void releaseRestoredHybridSnapshots(CacheRequestLease &lease)

Release hit-snapshot active pins after the stream-ordered restore has completed.

void retireHybridSnapshotReservation(

CacheRequestLease &lease,

HybridSnapshotReservation const &reservation

)

Release producer ownership after capture is terminal and publication has committed.

PublishResult publish(

CacheRequestLease &lease,

PublishRequest const &request

)

Commit ready full blocks after validating that the acquired prefix still describes this producer. EAGLE publication retains base and draft state through one common materialized boundary.

PublishResult publishHybrid(

CacheRequestLease &lease,

HybridPublishRequest const &request

)

Commit one already-captured exact hybrid checkpoint. The caller must make snapshot writes terminal first.

ResourcePools const &pools() const noexcept

BaseBlockIndex const &baseIndex() const noexcept

DraftPathIndex const &draftIndex() const noexcept

CacheRecordStore const &records() const noexcept

ContextCacheManagerMetrics const &metrics() const noexcept

struct PreparedPublication

Public Members

CacheRecord record

std::vector<PageId> canonicalBasePages

std::vector<MissingBaseMapping> missingBaseMappings

std::vector<ResourceId> cacheResources

std::optional<RecordId> recordLimitVictim

int32_t publishedBaseFullBlockCount = {}

struct AcquireResult

A plan is built and consumed under the manager’s single-writer contract, so it cannot become stale between planning and acquisition.

Public Members

std::optional<CacheRequestLease> lease

AcquireStatus status = {AcquireStatus::kAcquired}

ReusePlan plan

struct PublishRequest

Public Members

std::vector<BlockHash> fullBlockHashes

int32_t residentStateLength = {}

Greatest logical prefix ready for publication. For speculative leases, both base and draft state must be materialized through this boundary.

struct PublishResult

Detailed publication outcome describing the record path selected at commit time.

Public Members

PublishStatus status = {PublishStatus::kPublished}

std::optional<RecordId> record

std::vector<PageId> canonicalBasePages

Canonical base pages owned by the published record. This is never an active-row update recipe.

int32_t publishedBaseFullBlockCount = {}

Published boundary represented by canonicalBasePages; later producer pages remain unpublished.

struct HybridSnapshotReservation

Request-private snapshot slots reserved before the runtime enqueues a hybrid capture.

Public Members

int32_t recurrentSnapshotSlot = {}

std::optional<int32_t> partialKvSnapshotSlot

struct HybridPublishRequest

Public Members

std::vector<BlockHash> fullBlockHashes

HybridCheckpointKey checkpoint

HybridSnapshotReservation snapshots

struct ContextCacheManagerMetrics

Cumulative outcomes of record eviction. Reclaimed counts include only resources returned to a free list at the eviction point; resources still pinned by an active request are not counted.

Public Members

uint64_t evictedRecords = {}

uint64_t reclaimedBaseKvPages = {}

uint64_t reclaimedDraftKvPages = {}

uint64_t reclaimedRecurrentSnapshots = {}

uint64_t reclaimedPartialKvSnapshots = {}