Context Cache Coordinator#

class ContextCacheCoordinator#

Owns the complete host/device lifecycle around the CUDA-free ContextCacheManager.

The coordinator has no worker of its own; every call is made by the caller’s thread and callers must keep those calls serialized. Request handles may overlap in lifetime: several may be admitted and hold a page lease at once, which is the ownership shape in-flight batching needs.

Overlapping leases are not overlapping execution, and the distinction is load-bearing. Page-table rows bind to stable physical resident slots, while request-local reuse-length staging is reused across preparations. A prepare issued before the previous request’s async copy has landed would corrupt that staging buffer. A caller must therefore still finish one request’s steps before preparing the next; what a count rather than a flag permits is that the leases, not the execution, may interleave.

Normal publication occurs only at host-visible completion points already present in the runtime. An abnormal exit drains the bound stream before releasing active page references; a failed drain quarantines the entire request and poisons the coordinator until runtime-owned shutdown can establish quiescence. Poisoning gates new admissions, so a request already admitted when the stream failed is quarantined by its own drain rather than by the first failure.

Public Types

using StreamSynchronizer = std::function<cudaError_t(cudaStream_t)>#

Public Functions

ContextCacheCoordinator( ContextCacheConfig const &config, DeploymentConfig const &deployment, ContextCacheDeploymentProfile profile, ContextCachePhysicalResources resources, cudaStream_t stream, StreamSynchronizer synchronizer = {} )#

~ContextCacheCoordinator() noexcept#

ContextCacheCoordinator(ContextCacheCoordinator const&) = delete#

ContextCacheCoordinator &operator=( ContextCacheCoordinator const& ) = delete#

BeginRequestResult beginRequest( ContextCacheBatchAdmission const &admission, DecodingKvHeadroom const &headroom, cudaStream_t stream )#

AdmitSequenceResult admitSequence( RequestHandle &request, ContextCacheSequenceAdmission const &admission, DecodingKvHeadroom const &headroom )#

Join one more sequence to a live request: look up its prefix, lease its pages, and bind exactly its page-table row. Vanilla attention deployments only.

The device KV length for the new slot stays the runtime’s business — its seating sequence owns the cache-manager view — and the ledger is not advanced until finalizeSequenceAdmission reports the seated prefill’s outcome.

ContextCacheCoordinatorStatus finalizeSequenceAdmission( RequestHandle &request, int32_t slot, ContextCacheSequenceAdvance const &advance )#: The single-slot form of finalizePrefillPublication: record the seated prefill’s lookahead token, advance the slot’s committed prefix to the full input, and publish its ready full-block endpoints.

bool retractSequenceAdmission(RequestHandle &request) noexcept#

Undo the most recent admitSequence before its slot ever joined the runtime batch: release the lease, drop the sequence, and clear its page-table row.

Only legal while the admitted sequence is still the batch’s tail and the runtime holds no slot for it — the recovery path for a seating that threw between lease and slot append. A seated slot that failed later stays, terminal from birth, and leaves through the ordinary eviction instead.

ContextCacheCoordinatorStatus preparePrefill(RequestHandle &request)#: Bind every admitted row and reset logical cache lengths to the selected reuse boundaries.

ContextCacheCoordinatorStatus enqueuePrefillCaptures( RequestHandle &request )#: Reserve and enqueue hybrid prefill-end snapshots before the runtime’s existing prefill synchronization.

ContextCacheCoordinatorStatus finalizePrefillPublication( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const *commonStateLengths = nullptr )#: Apply the post-prefill sequence advance and publish every ready full-block endpoint. For EAGLE, commonStateLengths caps publication at the prefix materialized by both base and draft state.

ContextCacheCoordinatorStatus publishHybridMtpEndpoint( RequestHandle &request, int32_t slot, int32_t residentStateLength, Tensor const &baseHiddenStates, int32_t boundaryHiddenRow )#: Publish one Hybrid+MTP checkpoint at the stable predecessor boundary. This is the dedicated MTP publication entrypoint; the runtime drives it after the folded draft prefill has materialized the boundary draft state. It captures the recurrent state, the paired base+draft partial pages, and the successor-dependent boundary base-hidden row, then commits the exact checkpoint. Skipped for bypass, already-published, or empty prefixes.

ContextCacheCoordinatorStatus restoreHybridMtpBoundaryHidden( RequestHandle &request, int32_t slot, Tensor &baseHiddenStates, int32_t destinationRow )#: Restore the checkpoint’s saved boundary base-hidden row into baseHiddenStates[slot, destinationRow, :] for the runtime’s fold micro-forward. No synchronization: the caller orders this within its prefill stream.

ContextCacheCoordinatorStatus prepareDecodeStep( RequestHandle &request, DecodingKvHeadroom const &headroom )#: Grow and upload every row needed for the next decode working set before model execution.

ContextCacheCoordinatorStatus completeDecodeStep( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const &publishableCompletedSlots, std::vector<int32_t> const *commonStateLengths = nullptr )#: Apply the post-decode sequence advance after the decoder’s existing synchronization. For EAGLE, commonStateLengths excludes any unmaterialized accepted suffix and speculative lookahead.

ContextCacheCoordinatorStatus beginBatchCompaction( RequestHandle &request, std::vector<int32_t> const &oldToNew, int32_t newBatchSize, Tensor &deviceBatchMapping )#: Validate and upload the one authoritative old-to-new mapping before any old-slot retirement work.

ContextCacheCoordinatorStatus compactBatch(RequestHandle &request)#: Retire logical sequences while preserving survivor physical rows, then consume the eviction sync.

ContextCacheCoordinatorStatus finish(RequestHandle &request)#: Consume a normally completed request. This is idempotent for an already-empty handle.

ContextCacheCoordinatorStatus shutdown() noexcept#: Drain quarantined ownership before physical cache resources are destroyed.

ContextCacheMetrics metrics() const noexcept#

ContextCacheManager const &manager() const noexcept#

bool supportsLiveSequenceAdmission() const noexcept#: Whether this deployment can lease and bind a new sequence into an executing request.

struct AcquireSequenceResult#

Public Members

std::optional<CacheRequestLease> lease#

AcquireStatus status = {AcquireStatus::kInsufficientCapacity}#

ReusePlan plan#

bool forcedCold = {}#

struct AdmissionResult#

Public Members

RequestHandle request#

std::vector<int32_t> prefillStarts#: Per-sequence logical token offset at which runtime prefill begins.

struct AdmitSequenceResult#

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#

bool insufficientCapacity = {}#: True when the refusal is transient pool pressure rather than an error: pages free as resident sequences retire, so the caller may retry at a later step boundary.

int32_t prefillStart = {}#: Logical token offset at which the runtime’s prefill begins (== the reused prefix length).

class BaseEndpointPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy #

Public Functions

inline virtual char const *name() const noexcept override#

inline virtual void onPrefillFinalized( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const &publishableCompletedSlots, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-decode advance and publish endpoints for the completed slots.

inline explicit PublicationPolicy( ContextCacheCoordinator &coordinator ) noexcept#

struct BeginRequestResult#

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#

std::optional<AdmissionResult> admission#

class EagleSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy #

Public Functions

inline virtual char const *name() const noexcept override#

inline virtual void onPrefillFinalized( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const &publishableCompletedSlots, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual ContextCacheCoordinatorStatus onTerminalize( RequestHandle &request ) override#: EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.

inline explicit PublicationPolicy( ContextCacheCoordinator &coordinator ) noexcept#

class HybridMtpPolicy : public trt_edgellm::rt::ContextCacheCoordinator::HybridSnapshotPolicy #

Public Functions

inline virtual char const *name() const noexcept override#

inline virtual ContextCacheCoordinatorStatus publishMtpBoundary( RequestHandle &request, int32_t slot, int32_t residentStateLength, Tensor const &baseHiddenStates, int32_t boundaryHiddenRow ) override#: Hybrid+MTP-only: publish the successor-boundary checkpoint. Unreachable for other flavors.

inline virtual ContextCacheCoordinatorStatus restoreMtpBoundary( RequestHandle &request, int32_t slot, Tensor &baseHiddenStates, int32_t destinationRow ) override#: Hybrid+MTP-only: restore the checkpoint’s saved boundary hidden row. Unreachable for other flavors.

class HybridSnapshotPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy #

Subclassed by trt_edgellm::rt::ContextCacheCoordinator::HybridMtpPolicy

Public Functions

inline virtual char const *name() const noexcept override#

inline virtual void onPrefillFinalized( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const &publishableCompletedSlots, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-decode advance and publish endpoints for the completed slots.

inline explicit PublicationPolicy( ContextCacheCoordinator &coordinator ) noexcept#

class PublicationPolicy#

Subclassed by trt_edgellm::rt::ContextCacheCoordinator::BaseEndpointPolicy, trt_edgellm::rt::ContextCacheCoordinator::EagleSpecPolicy, trt_edgellm::rt::ContextCacheCoordinator::HybridSnapshotPolicy, trt_edgellm::rt::ContextCacheCoordinator::SharedKvSpecPolicy

Public Functions

inline explicit PublicationPolicy( ContextCacheCoordinator &coordinator ) noexcept#

virtual ~PublicationPolicy() noexcept = default#

virtual char const *name() const noexcept = 0#

virtual void onPrefillFinalized( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const *commonStateLengths ) = 0#: Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

virtual ContextCacheCoordinatorStatus onDecodeCompleted( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const &publishableCompletedSlots, std::vector<int32_t> const *commonStateLengths ) = 0#: Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual ContextCacheCoordinatorStatus onTerminalize( RequestHandle& )#: EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.

inline virtual ContextCacheCoordinatorStatus publishMtpBoundary( RequestHandle&, int32_t, int32_t, Tensor const&, int32_t )#: Hybrid+MTP-only: publish the successor-boundary checkpoint. Unreachable for other flavors.

inline virtual ContextCacheCoordinatorStatus restoreMtpBoundary( RequestHandle&, int32_t, Tensor&, int32_t )#: Hybrid+MTP-only: restore the checkpoint’s saved boundary hidden row. Unreachable for other flavors.

class RequestHandle#

Public Functions

RequestHandle(RequestHandle &&other) noexcept#

RequestHandle &operator=(RequestHandle &&other) = delete#

RequestHandle(RequestHandle const&) = delete#

RequestHandle &operator=(RequestHandle const&) = delete#

~RequestHandle() noexcept#

bool valid() const noexcept#

struct Impl#

Public Types

enum class Phase : uint8_t#

Values:

enumerator kAdmitted#

enumerator kExecuting#

enumerator kFinishing#

Public Functions

inline explicit Impl( ContextCacheCoordinator &coordinator, cudaStream_t requestStream ) noexcept#

inline bool admitted() const noexcept#

inline bool executing() const noexcept#

inline bool hasPendingDeviceWork() const noexcept#

inline bool awaitingFirstSpecCompletion() const noexcept#

inline void beginPrefill() noexcept#: kAdmitted -> kExecuting with the prefill work enqueued on the stream.

inline void markDeviceWorkEnqueued() noexcept#: Fresh GPU work was enqueued on impl.stream (prefill grow, decode grow, fold, compaction copy).

inline void markDeviceWorkSynchronized() noexcept#: impl.stream was explicitly synchronized (or a verification round already synchronized it).

inline void markDeviceWorkResolvedWithoutSync() noexcept#: The enqueued work is provably terminal without a coordinator synchronize (no async publish work remains).

inline void beginSpecDraftInit() noexcept#: EAGLE only: the ordered draft initialization awaits its first verification round before publishing.

inline void endSpecDraftInit() noexcept#

inline void markFinishing() noexcept#: Enter the terminal phase (normal completion at empty batch, or an unrecoverable capacity/growth failure).

Public Members

RequestSlotToken requestSlot#: Declared before sequences so its destructor drops the active-request count only after every lease is released.

std::unique_ptr<PublicationPolicy> publicationPolicy#

ContextCacheCoordinator *owner = {}#

cudaStream_t stream = {}#

bool speculativeRequest = {}#

DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}#

std::vector<int32_t> pendingCompactionMapping#

int32_t pendingCompactionBatchSize = {-1}#

Tensor const *pendingDeviceBatchMapping = {}#

std::vector<SequenceState> sequences#

struct RequestSlotToken#

Public Functions

inline explicit RequestSlotToken( ContextCacheCoordinator &coordinator ) noexcept#

inline RequestSlotToken(RequestSlotToken &&other) noexcept#

RequestSlotToken &operator=(RequestSlotToken&&) = delete#

RequestSlotToken(RequestSlotToken const&) = delete#

RequestSlotToken &operator=(RequestSlotToken const&) = delete#

inline ~RequestSlotToken() noexcept#

Public Members

ContextCacheCoordinator *owner = {}#

struct SequenceState#

Public Members

CacheRequestLease lease#

ResidentRef resident#

std::vector<int32_t> tokenIds#

BlockKeyExtras keyExtras#

std::vector<Hash128> perPositionMediaHash#

int32_t reuseTokenLength = {}#

ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}#

ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}#

int32_t replayTailLength = {}#

int32_t committedStateLength = {}#

int32_t publishedFullBlockCount = {}#

int32_t publishedExactLength = {}#

int32_t commonStateLength = {}#

std::optional<int32_t> frozenSpecPrefillLength#

std::optional<StagedHybridPublication> stagedHybridPublication#

struct StagedHybridPublication#

Public Members

HybridCheckpointKey checkpoint#

HybridSnapshotReservation snapshots#

PublicationPoint point = {}#

class SharedKvSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy #

Public Functions

inline virtual char const *name() const noexcept override#

inline virtual void onPrefillFinalized( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted( RequestHandle &request, std::vector<ContextCacheSequenceAdvance> const &advances, std::vector<int32_t> const &publishableCompletedSlots, std::vector<int32_t> const *commonStateLengths ) override#: Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual ContextCacheCoordinatorStatus onTerminalize( RequestHandle &request ) override#: EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.

inline explicit PublicationPolicy( ContextCacheCoordinator &coordinator ) noexcept#

class RequestHandle

Public Functions

RequestHandle(RequestHandle &&other) noexcept

RequestHandle &operator=(RequestHandle &&other) = delete

RequestHandle(RequestHandle const&) = delete

RequestHandle &operator=(RequestHandle const&) = delete

~RequestHandle() noexcept

bool valid() const noexcept

struct Impl

Public Types

enum class Phase : uint8_t

Values:

enumerator kAdmitted

enumerator kExecuting

enumerator kFinishing

Public Functions

inline explicit Impl( ContextCacheCoordinator &coordinator, cudaStream_t requestStream ) noexcept

inline bool admitted() const noexcept

inline bool executing() const noexcept

inline bool hasPendingDeviceWork() const noexcept

inline bool awaitingFirstSpecCompletion() const noexcept

inline void beginPrefill() noexcept: kAdmitted -> kExecuting with the prefill work enqueued on the stream.

inline void markDeviceWorkEnqueued() noexcept: Fresh GPU work was enqueued on impl.stream (prefill grow, decode grow, fold, compaction copy).

inline void markDeviceWorkSynchronized() noexcept: impl.stream was explicitly synchronized (or a verification round already synchronized it).

inline void markDeviceWorkResolvedWithoutSync() noexcept: The enqueued work is provably terminal without a coordinator synchronize (no async publish work remains).

inline void beginSpecDraftInit() noexcept: EAGLE only: the ordered draft initialization awaits its first verification round before publishing.

inline void endSpecDraftInit() noexcept

inline void markFinishing() noexcept: Enter the terminal phase (normal completion at empty batch, or an unrecoverable capacity/growth failure).

Public Members

RequestSlotToken requestSlot: Declared before sequences so its destructor drops the active-request count only after every lease is released.

std::unique_ptr<PublicationPolicy> publicationPolicy

ContextCacheCoordinator *owner = {}

cudaStream_t stream = {}

bool speculativeRequest = {}

DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}

std::vector<int32_t> pendingCompactionMapping

int32_t pendingCompactionBatchSize = {-1}

Tensor const *pendingDeviceBatchMapping = {}

std::vector<SequenceState> sequences

struct RequestSlotToken

Public Functions

inline explicit RequestSlotToken( ContextCacheCoordinator &coordinator ) noexcept

inline RequestSlotToken(RequestSlotToken &&other) noexcept

RequestSlotToken &operator=(RequestSlotToken&&) = delete

RequestSlotToken(RequestSlotToken const&) = delete

RequestSlotToken &operator=(RequestSlotToken const&) = delete

inline ~RequestSlotToken() noexcept

Public Members

ContextCacheCoordinator *owner = {}

struct SequenceState

Public Members

CacheRequestLease lease

ResidentRef resident

std::vector<int32_t> tokenIds

BlockKeyExtras keyExtras

std::vector<Hash128> perPositionMediaHash

int32_t reuseTokenLength = {}

ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}

ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}

int32_t replayTailLength = {}

int32_t committedStateLength = {}

int32_t publishedFullBlockCount = {}

int32_t publishedExactLength = {}

int32_t commonStateLength = {}

std::optional<int32_t> frozenSpecPrefillLength

std::optional<StagedHybridPublication> stagedHybridPublication

struct StagedHybridPublication

Public Members

HybridCheckpointKey checkpoint

HybridSnapshotReservation snapshots

PublicationPoint point = {}

struct ContextCachePhysicalResources#

Validated physical resources borrowed from SharedResources for the coordinator lifetime.

Public Members

HybridCacheManager &baseCache#

KVPageTable &basePageTable#

HybridCacheManager *draftCache = {}#

KVPageTable *draftPageTable = {}#

struct ContextCacheSequenceAdmission#

One complete logical input admitted to the cache before its executable suffix is derived.

Public Members

std::vector<int32_t> tokenIds#

BlockKeyExtras keyExtras#: Request-wide non-token identity; LoRA/isolation identity is constant for the sequence.

std::vector<Hash128> perPositionMediaHash#: Per-position media content hash. Empty means text-only. When non-empty, must have tokenIds.size() entries. A non-zero Hash128 at position i causes the block hash to consume that 128-bit digest instead of the token ID.

ResidentRef resident#: Scheduler-owned physical state identity; the coordinator may not derive this from logical sequence order.

std::optional<ContextCacheLookupPolicy> lookupPolicy#: Optional sequence-level override of ContextCacheBatchAdmission::lookupPolicy.

struct ContextCacheBatchAdmission#

One serialized runtime request. Bypass still uses managed private pages but neither looks up nor publishes state.

Public Members

std::vector<ContextCacheSequenceAdmission> sequences#

bool speculativeRequest = {}#

DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}#

ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}#

ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}#

int32_t replayTailLength = {0}#: Hybrid+MTP replay tail length derived before cache admission.

struct ContextCacheSequenceAdvance#

Host-visible sequence advance observed after an existing stream synchronization.

Public Members

int32_t const *acceptedTokenIds = {}#

int32_t acceptedTokenCount = {}#

int32_t committedStateLength = {}#: Greatest logical token boundary whose model state is materialized in the bound cache.

Public Static Attributes

static int32_t kHoldCommittedStateLength = -1#: committedStateLength value for a zero advance that holds the ledger’s current committed length in place. A concrete length with no accepted tokens represents a state-only advance.

struct AdmissionResult

Public Members

RequestHandle request

std::vector<int32_t> prefillStarts: Per-sequence logical token offset at which runtime prefill begins.

struct BeginRequestResult

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}

std::optional<AdmissionResult> admission

struct AdmitSequenceResult

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}

bool insufficientCapacity = {}: True when the refusal is transient pool pressure rather than an error: pages free as resident sequences retire, so the caller may retry at a later step boundary.

int32_t prefillStart = {}: Logical token offset at which the runtime’s prefill begins (== the reused prefix length).