Context Cache Coordinator#

class ContextCacheCoordinator#

Owns the complete host/device lifecycle around the CUDA-free ContextCacheManager.

The coordinator has no worker of its own; every call is made by the caller’s thread and callers must keep those calls serialized. Request handles may overlap in lifetime: several may be admitted and hold a page lease at once, which is the ownership shape in-flight batching needs.

Overlapping leases are not overlapping execution, and the distinction is load-bearing. Page-table rows bind to stable physical resident slots, while request-local reuse-length staging is reused across preparations. A prepare issued before the previous request’s async copy has landed would corrupt that staging buffer. A caller must therefore still finish one request’s steps before preparing the next; what a count rather than a flag permits is that the leases, not the execution, may interleave.

Normal publication occurs only at host-visible completion points already present in the runtime. An abnormal exit drains the bound stream before releasing active page references; a failed drain quarantines the entire request and poisons the coordinator until runtime-owned shutdown can establish quiescence. Poisoning gates new admissions, so a request already admitted when the stream failed is quarantined by its own drain rather than by the first failure.

Sequence-admission transaction

Every sequence joined through admitSequence is in admission until finalizeSequenceAdmission commits its final prefill chunk or abortSequenceAdmission discards it. Any number of sequences may be in admission at once, at any execution index, and each tracks its own in-flight device work. Per chunk the caller runs prepare -> (final chunk only: capture) -> its own stream synchronization -> complete -> advance (non-final) or finalize (final).

ContextCacheCoordinatorStatus prepareSequenceAdmission(
RequestHandle &request,
int32_t sequenceIndex
)#

Open the next prefill chunk of a sequence in admission. Before the first chunk this restores the sequence’s recurrent and partial-KV checkpoint into its stable resident slot without touching any other resident sequence. sequenceIndex is the transient execution index.

ContextCacheCoordinatorStatus enqueueSequenceAdmissionCapture(
RequestHandle &request,
int32_t sequenceIndex
)#

Reserve and enqueue the sequence’s prefill-end snapshot after its final chunk and before the runtime’s existing sampled-token synchronization. This is a no-op for non-hybrid deployments.

ContextCacheCoordinatorStatus completeSequenceAdmissionDeviceWork(
RequestHandle &request,
int32_t sequenceIndex
)#

Resolve the sequence’s open chunk after the runtime’s stream synchronization. Other sequences’ chunks stay open until their own completion.

ContextCacheCoordinatorStatus advanceSequenceAdmission(
RequestHandle &request,
int32_t sequenceIndex,
int32_t committedStateLength
)#

Record a committed non-final prefill chunk: the slot’s state now covers the first committedStateLength prompt tokens. Nothing is published and no snapshot is captured.

ContextCacheCoordinatorStatus abortSequenceAdmission(
RequestHandle &request,
int32_t sequenceIndex
)#

Discard a failed or cancelled sequence-admission transaction so ordinary eviction can retire it; the eviction’s compaction releases its lease. Drains the stream only when the sequence has an open chunk. The sequence never publishes afterwards. Throws for a sequence that is not in admission (finalized, aborted, or admitted with the request).

ContextCacheCoordinatorStatus finalizeSequenceAdmission(
RequestHandle &request,
int32_t sequenceIndex,
ContextCacheSequenceAdvance const &advance,
std::optional<int32_t> commonStateLength = std::nullopt
)#

The single-slot form of finalizePrefillPublication: record the final chunk’s lookahead token, advance the slot’s committed prefix to the full input, and publish its ready prefix state (the hybrid snapshot captured for this sequence only).

commonStateLength is the prefix a speculative sequence’s draft has materialized (the full input when empty). A Hybrid+spec sequence published its checkpoint during the prefill (publishHybridSpecEndpoint); an EAGLE-flavor sequence’s prompt endpoint publishes at the next decode completion, capped at the common state.

Public Types

using StreamSynchronizer = std::function<cudaError_t(cudaStream_t)>#

Public Functions

ContextCacheCoordinator(
ContextCacheConfig const &config,
DeploymentConfig const &deployment,
ContextCacheDeploymentProfile profile,
ContextCachePhysicalResources resources,
cudaStream_t stream,
StreamSynchronizer synchronizer = {}
)#
~ContextCacheCoordinator() noexcept#
ContextCacheCoordinator(ContextCacheCoordinator const&) = delete#
ContextCacheCoordinator &operator=(
ContextCacheCoordinator const&
) = delete#
BeginRequestResult beginRequest(
ContextCacheBatchAdmission const &admission,
DecodingKvHeadroom const &headroom,
cudaStream_t stream
)#
AdmitSequenceResult admitSequence(
RequestHandle &request,
ContextCacheSequenceAdmission const &admission,
DecodingKvHeadroom const &headroom
)#

Join one more sequence to a live request: look up its prefix, lease its pages, and bind exactly its page-table row (and its draft page-table row, for a paired speculative lease). Attention deployments without bounded SWA only.

The request may hold no sequence yet: an empty request admitted with no sequences takes its first one here, through the same path as every later one. The sequence inherits the request’s lookup, commit and replay-tail policies unless its admission overrides the lookup policy.

The device KV length for the new slot stays the runtime’s business – its seating sequence owns the cache-manager view – and the ledger is not advanced until advanceSequenceAdmission or finalizeSequenceAdmission reports a committed prefill chunk.

Other sequences may still be in admission, and their chunk work may be in flight: the lease comes from pages no enqueued work references, and the row upload is stream-ordered for a slot no enqueued step reads. Request-wide work (a decode step or a compaction) must be resolved first.

bool retractSequenceAdmission(RequestHandle &request) noexcept#

Undo the most recent admitSequence before its slot ever joined the runtime batch: release the lease, drop the sequence, and clear its page-table row.

Only legal while the admitted sequence is still the batch’s tail and the runtime holds no slot for it – the recovery path for a seating that threw between lease and slot append. A seated slot that failed later stays, terminal from birth, and leaves through the ordinary eviction instead. Retracting from a request that holds no sequence poisons the coordinator: nothing was admitted, so the caller’s bookkeeping is already wrong.

ContextCacheCoordinatorStatus preparePrefill(RequestHandle &request)#

Bind every admitted row and reset logical cache lengths to the selected reuse boundaries, entering execution. A request admitted with no sequences has no prefill to complete and is settled at once.

ContextCacheCoordinatorStatus enqueuePrefillCaptures(
RequestHandle &request
)#

Reserve and enqueue hybrid prefill-end snapshots before the runtime’s existing prefill synchronization.

ContextCacheCoordinatorStatus finalizePrefillPublication(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths = nullptr,
SwaWindowStateView const *swaWindowStates = nullptr
)#

Apply the post-prefill sequence advance and publish every ready full-block endpoint. For EAGLE, commonStateLengths caps publication at the prefix materialized by both base and draft state.

ContextCacheCoordinatorStatus publishHybridSpecEndpoint(
RequestHandle &request,
int32_t slot,
int32_t residentStateLength,
Tensor const &baseHiddenStates,
int32_t boundaryHiddenRow
)#

Publish one Hybrid+spec checkpoint at the stable predecessor boundary. This is the dedicated MTP publication entrypoint; the runtime drives it after the folded draft prefill has materialized the boundary draft state, for a blocking request’s sequence or one in sequence admission. It captures the recurrent state, the paired base+draft partial pages, and the successor-dependent boundary base-hidden row, then commits the exact checkpoint. Skipped for bypass, already-published, or empty prefixes.

ContextCacheCoordinatorStatus restoreHybridSpecBoundaryHidden(
RequestHandle &request,
int32_t slot,
Tensor &baseHiddenStates,
int32_t destinationRow
)#

Restore the checkpoint’s saved boundary base-hidden row into baseHiddenStates[slot, destinationRow, :] for the runtime’s fold micro-forward. No synchronization: the caller orders this within its prefill stream.

ContextCacheCoordinatorStatus prepareDecodeStep(
RequestHandle &request,
DecodingKvHeadroom const &headroom,
std::vector<int32_t> const *extraTokensPerSlot = nullptr,
std::vector<int32_t> const *draftExtraTokensPerSlot = nullptr
)#

Grow and upload every row needed for the next decode working set before model execution. extraTokensPerSlot, when set, is each sequence’s own growth beyond its committed state (one, or the pending run a jump-forward left; a jump-forward slot’s wider verify round) and replaces headroom.baseExtraTokens. draftExtraTokensPerSlot likewise replaces headroom.draftExtraTokens; paired speculative requests only.

ContextCacheCoordinatorStatus prepareDecodeStep(
RequestHandle &request,
DecodingKvHeadroom const &headroom,
std::vector<int32_t> const &decodeSequences
)#

Grow only the rows of decodeSequences (execution indices, each with a complete prompt) for a step in which the other sequences do not decode. Every other sequence must carry the hold advance in the matching completeDecodeStep. Non-speculative requests only.

ContextCacheCoordinatorStatus completeDecodeStep(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths = nullptr,
SwaWindowStateView const *swaWindowStates = nullptr
)#

Apply the post-decode sequence advance after the decoder’s existing synchronization. A sequence that did not decode this step – outside the prepared decode subset, or still in admission – must carry the hold advance and cannot be published. For EAGLE, commonStateLengths excludes any unmaterialized accepted suffix and speculative lookahead.

bool isExternallyOwnedSwaPage(
RequestHandle const &request,
int32_t slot,
PageId page
) const#
PageId replaceSwaPage(
RequestHandle &request,
int32_t slot,
PageId page
)#
ContextCacheCoordinatorStatus beginBatchCompaction(
RequestHandle &request,
std::vector<int32_t> const &oldToNew,
int32_t newBatchSize,
Tensor &deviceBatchMapping
)#

Validate and upload the one authoritative old-to-new mapping before any old-slot retirement work.

ContextCacheCoordinatorStatus compactBatch(RequestHandle &request)#

Retire logical sequences while preserving survivor physical rows, then consume the eviction sync.

ContextCacheCoordinatorStatus finish(RequestHandle &request)#

Consume a normally completed request. This is idempotent for an already-empty handle.

ContextCacheCoordinatorStatus shutdown() noexcept#

Drain quarantined ownership before physical cache resources are destroyed.

ContextCacheMetrics metrics() const noexcept#
ContextCacheManager const &manager() const noexcept#
ContextCacheManager &manager() noexcept#
bool supportsLiveSequenceAdmission() const noexcept#

Whether this deployment can lease and bind a new sequence into an executing request.

int32_t maxPrefillLength() const noexcept#

The longest uncached suffix an admitted sequence may leave to prefill: the engine’s maximum input length.

int32_t activeRequests() const noexcept#

Requests begun and not yet finished or abandoned, quarantined ones included.

struct AcquireSequenceResult#

Public Members

std::optional<CacheRequestLease> lease#
AcquireStatus status = {AcquireStatus::kInsufficientCapacity}#
ReusePlan plan#
bool forcedCold = {}#
std::optional<int32_t> overlongPrefillLength = {}#

Set, with no lease, when the planned uncached suffix exceeds mMaxPrefillLength.

struct AdmissionResult#

Public Members

RequestHandle request#
std::vector<int32_t> prefillStarts#

Per-sequence logical token offset at which runtime prefill begins.

std::vector<SwaKVCacheState> swaStates#

Seeded states whose page references remain owned by the corresponding context-cache leases.

struct AdmitSequenceResult#

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#
bool insufficientCapacity = {}#

True when the refusal is transient pool pressure rather than an error: pages free as resident sequences retire, so the caller may retry at a later step boundary.

int32_t prefillStart = {}#

Logical token offset at which the runtime’s prefill begins (== the reused prefix length); on an overlongPrefill refusal, the reused length the lookup found.

bool overlongPrefill = {}#

Refused because the uncached suffix exceeds maxPrefillLength(); nothing was leased or evicted.

class BaseEndpointPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#

Public Functions

inline virtual char const *name() const noexcept override#
inline virtual void onPrefillFinalized(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) override#

Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) override#

Apply the post-decode advance and publish endpoints for the completed slots.

inline explicit PublicationPolicy(
ContextCacheCoordinator &coordinator
) noexcept#
struct BeginRequestResult#

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#
std::optional<AdmissionResult> admission#
std::optional<int32_t> overlongPrefillLength = {}#

Set on a refusal whose cause is a sequence’s uncached suffix exceeding maxPrefillLength(): that suffix length. Such a sequence is refused before anything is leased, so no record is evicted for it.

class EagleSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#

Public Functions

inline virtual char const *name() const noexcept override#
inline virtual void onPrefillFinalized(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const*
) override#

Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const*
) override#

Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual void onSequenceAdmissionFinalized(
RequestHandle &request,
int32_t sequenceIndex
) override#

A sequence admitted into the executing request seated its draft prefill without a synchronization: its prompt endpoint freezes now and publishes, capped at the common state, at the next decode completion (whose verification synchronizes) or at terminalization.

inline virtual ContextCacheCoordinatorStatus onTerminalize(
RequestHandle &request
) override#

EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.

inline virtual ContextCacheCoordinatorStatus onCompaction(
RequestHandle &request,
std::vector<int32_t> const &oldToNew
) override#

A seated sequence that survives the compaction keeps its frozen endpoint for its first decode completion; only a retiring one publishes now.

inline explicit PublicationPolicy(
ContextCacheCoordinator &coordinator
) noexcept#
class HybridSnapshotPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#

Subclassed by trt_edgellm::rt::ContextCacheCoordinator::HybridSpecPolicy

Public Functions

inline virtual char const *name() const noexcept override#
inline virtual void onPrefillFinalized(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const*
) override#

Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const*
) override#

Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual void onSequenceAdmissionFinalized(
RequestHandle &request,
int32_t sequenceIndex
) override#

A sequence admitted into the executing request committed its prompt: publish (or arm) its prefill-end endpoint. The default publishes the ready full-block base endpoint.

inline explicit PublicationPolicy(
ContextCacheCoordinator &coordinator
) noexcept#
class HybridSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::HybridSnapshotPolicy#

Public Functions

inline virtual char const *name() const noexcept override#
inline virtual void onPrefillFinalized(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const*
) override#

Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) override#

Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual void onSequenceAdmissionFinalized(
RequestHandle &request,
int32_t sequenceIndex
) override#

The checkpoint was published during the prefill (publishMtpBoundary), once the draft had seated the prompt.

inline virtual ContextCacheCoordinatorStatus publishMtpBoundary(
RequestHandle &request,
int32_t slot,
int32_t residentStateLength,
Tensor const &baseHiddenStates,
int32_t boundaryHiddenRow
) override#

Hybrid+spec-only: publish the successor-boundary checkpoint. Unreachable for other flavors.

inline virtual ContextCacheCoordinatorStatus restoreMtpBoundary(
RequestHandle &request,
int32_t slot,
Tensor &baseHiddenStates,
int32_t destinationRow
) override#

Hybrid+spec-only: restore the checkpoint’s saved boundary hidden row. Unreachable for other flavors.

class PublicationPolicy#

Subclassed by trt_edgellm::rt::ContextCacheCoordinator::BaseEndpointPolicy, trt_edgellm::rt::ContextCacheCoordinator::EagleSpecPolicy, trt_edgellm::rt::ContextCacheCoordinator::HybridSnapshotPolicy, trt_edgellm::rt::ContextCacheCoordinator::SharedKvSpecPolicy

Public Functions

inline explicit PublicationPolicy(
ContextCacheCoordinator &coordinator
) noexcept#
virtual ~PublicationPolicy() noexcept = default#
virtual char const *name() const noexcept = 0#
virtual void onPrefillFinalized(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) = 0#

Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

virtual ContextCacheCoordinatorStatus onDecodeCompleted(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) = 0#

Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual void onSequenceAdmissionFinalized(
RequestHandle &request,
int32_t sequenceIndex
)#

A sequence admitted into the executing request committed its prompt: publish (or arm) its prefill-end endpoint. The default publishes the ready full-block base endpoint.

inline virtual ContextCacheCoordinatorStatus onTerminalize(
RequestHandle&
)#

EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.

inline virtual ContextCacheCoordinatorStatus onCompaction(
RequestHandle &request,
std::vector<int32_t> const&
)#

Before a compaction that retires the rows oldToNew maps below zero. Default: terminalize.

inline virtual ContextCacheCoordinatorStatus publishMtpBoundary(
RequestHandle&,
int32_t,
int32_t,
Tensor const&,
int32_t
)#

Hybrid+spec-only: publish the successor-boundary checkpoint. Unreachable for other flavors.

inline virtual ContextCacheCoordinatorStatus restoreMtpBoundary(
RequestHandle&,
int32_t,
Tensor&,
int32_t
)#

Hybrid+spec-only: restore the checkpoint’s saved boundary hidden row. Unreachable for other flavors.

class RequestHandle#

Public Functions

RequestHandle(RequestHandle &&other) noexcept#
RequestHandle &operator=(RequestHandle &&other) = delete#
RequestHandle(RequestHandle const&) = delete#
RequestHandle &operator=(RequestHandle const&) = delete#
~RequestHandle() noexcept#
bool valid() const noexcept#
struct Impl#

Public Types

enum class Phase : uint8_t#

Values:

enumerator kAdmitted#
enumerator kExecuting#
enumerator kFinishing#
enum class AdmissionState : uint8_t#

Sequences admitted with the request start settled. A live admission stays kAdmitting until its final prefill chunk commits; an aborted one never publishes and waits for eviction.

Values:

enumerator kSettled#
enumerator kAdmitting#
enumerator kAborted#

Public Functions

inline explicit Impl(
ContextCacheCoordinator &coordinator,
cudaStream_t requestStream
) noexcept#
inline bool admitted() const noexcept#
inline bool executing() const noexcept#
inline bool hasPendingDeviceWork() const noexcept#
inline bool hasPendingRequestWork() const noexcept#
inline bool hasUnsynchronizedSeatedPrefill() const noexcept#

A sequence seated into the executing request froze its prompt endpoint over a draft prefill that nothing has synchronized yet: it is synchronized by the next decode verification or at terminalization.

inline bool hasOpenAdmissionChunk() const noexcept#
inline bool decodesThisStep(size_t sequenceIndex) const noexcept#
inline SequenceState &sequenceInAdmission(
int32_t sequenceIndex,
char const *operation
)#
inline bool awaitingFirstSpecCompletion() const noexcept#
inline void beginPrefill() noexcept#

kAdmitted -> kExecuting with the prefill work enqueued on the stream.

inline void markDeviceWorkEnqueued() noexcept#

Fresh GPU work was enqueued on impl.stream (prefill grow, decode grow, fold, compaction copy).

inline void markDeviceWorkSynchronized() noexcept#

impl.stream was explicitly synchronized (or a verification round already synchronized it).

inline void markDeviceWorkResolvedWithoutSync() noexcept#

The enqueued work is provably terminal without a coordinator synchronize (no async publish work remains).

inline void beginSpecDraftInit() noexcept#

EAGLE only: the ordered draft initialization awaits its first verification round before publishing.

inline void endSpecDraftInit() noexcept#
inline void markFinishing() noexcept#

Enter the terminal phase after an unrecoverable capacity/growth failure. Normal completion is the caller’s finish(); a batch that drained to nothing stays executing until then.

Public Members

RequestSlotToken requestSlot#

Declared before sequences so its destructor drops the active-request count only after every lease is released.

std::unique_ptr<PublicationPolicy> publicationPolicy#
ContextCacheCoordinator *owner = {}#
cudaStream_t stream = {}#
bool speculativeRequest = {}#
DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}#
std::vector<int32_t> pendingCompactionMapping#
int32_t pendingCompactionBatchSize = {-1}#
Tensor const *pendingDeviceBatchMapping = {}#
std::vector<SequenceState> sequences#
std::vector<uint8_t> decodeParticipants#

Execution indices that decode in the prepared step; empty means every sequence.

struct RequestSlotToken#

Public Functions

inline explicit RequestSlotToken(
ContextCacheCoordinator &coordinator
) noexcept#
inline RequestSlotToken(RequestSlotToken &&other) noexcept#
RequestSlotToken &operator=(RequestSlotToken&&) = delete#
RequestSlotToken(RequestSlotToken const&) = delete#
RequestSlotToken &operator=(RequestSlotToken const&) = delete#
inline ~RequestSlotToken() noexcept#

Public Members

ContextCacheCoordinator *owner = {}#
struct SequenceState#

Public Members

CacheRequestLease lease#
AdmissionState admissionState = {AdmissionState::kSettled}#
bool admissionCheckpointRestored = {}#
bool admissionChunkOpen = {}#

A prepared prefill chunk whose device work the caller has not yet reported complete.

ResidentRef resident#
std::vector<int32_t> tokenIds#
BlockKeyExtras keyExtras#
std::vector<Hash128> perPositionMediaHash#
int32_t reuseTokenLength = {}#
ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}#
ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}#
int32_t replayTailLength = {}#
int32_t committedStateLength = {}#
int32_t publishedFullBlockCount = {}#
int32_t publishedExactLength = {}#
int32_t commonStateLength = {}#
std::optional<int32_t> frozenSpecPrefillLength#
std::optional<StagedHybridPublication> stagedHybridPublication#
struct StagedHybridPublication#

Public Members

HybridCheckpointKey checkpoint#
HybridSnapshotReservation snapshots#
PublicationPoint point = {}#
class SharedKvSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#

Public Functions

inline virtual char const *name() const noexcept override#
inline virtual void onPrefillFinalized(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) override#

Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.

inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
RequestHandle &request,
std::vector<ContextCacheSequenceAdvance> const &advances,
std::vector<int32_t> const &publishableCompletedSlots,
std::vector<int32_t> const *commonStateLengths,
SwaWindowStateView const *swaWindowStates
) override#

Apply the post-decode advance and publish endpoints for the completed slots.

inline virtual void onSequenceAdmissionFinalized(
RequestHandle &request,
int32_t sequenceIndex
) override#

A sequence admitted into the executing request committed its prompt: publish (or arm) its prefill-end endpoint. The default publishes the ready full-block base endpoint.

inline virtual ContextCacheCoordinatorStatus onTerminalize(
RequestHandle &request
) override#

EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.

inline virtual ContextCacheCoordinatorStatus onCompaction(
RequestHandle &request,
std::vector<int32_t> const &oldToNew
) override#

Before a compaction that retires the rows oldToNew maps below zero. Default: terminalize.

inline explicit PublicationPolicy(
ContextCacheCoordinator &coordinator
) noexcept#
class RequestHandle

Public Functions

RequestHandle(RequestHandle &&other) noexcept
RequestHandle &operator=(RequestHandle &&other) = delete
RequestHandle(RequestHandle const&) = delete
RequestHandle &operator=(RequestHandle const&) = delete
~RequestHandle() noexcept
bool valid() const noexcept
struct Impl

Public Types

enum class Phase : uint8_t

Values:

enumerator kAdmitted
enumerator kExecuting
enumerator kFinishing
enum class AdmissionState : uint8_t

Sequences admitted with the request start settled. A live admission stays kAdmitting until its final prefill chunk commits; an aborted one never publishes and waits for eviction.

Values:

enumerator kSettled
enumerator kAdmitting
enumerator kAborted

Public Functions

inline explicit Impl(
ContextCacheCoordinator &coordinator,
cudaStream_t requestStream
) noexcept
inline bool admitted() const noexcept
inline bool executing() const noexcept
inline bool hasPendingDeviceWork() const noexcept
inline bool hasPendingRequestWork() const noexcept
inline bool hasUnsynchronizedSeatedPrefill() const noexcept

A sequence seated into the executing request froze its prompt endpoint over a draft prefill that nothing has synchronized yet: it is synchronized by the next decode verification or at terminalization.

inline bool hasOpenAdmissionChunk() const noexcept
inline bool decodesThisStep(size_t sequenceIndex) const noexcept
inline SequenceState &sequenceInAdmission(
int32_t sequenceIndex,
char const *operation
)
inline bool awaitingFirstSpecCompletion() const noexcept
inline void beginPrefill() noexcept

kAdmitted -> kExecuting with the prefill work enqueued on the stream.

inline void markDeviceWorkEnqueued() noexcept

Fresh GPU work was enqueued on impl.stream (prefill grow, decode grow, fold, compaction copy).

inline void markDeviceWorkSynchronized() noexcept

impl.stream was explicitly synchronized (or a verification round already synchronized it).

inline void markDeviceWorkResolvedWithoutSync() noexcept

The enqueued work is provably terminal without a coordinator synchronize (no async publish work remains).

inline void beginSpecDraftInit() noexcept

EAGLE only: the ordered draft initialization awaits its first verification round before publishing.

inline void endSpecDraftInit() noexcept
inline void markFinishing() noexcept

Enter the terminal phase after an unrecoverable capacity/growth failure. Normal completion is the caller’s finish(); a batch that drained to nothing stays executing until then.

Public Members

RequestSlotToken requestSlot

Declared before sequences so its destructor drops the active-request count only after every lease is released.

std::unique_ptr<PublicationPolicy> publicationPolicy
ContextCacheCoordinator *owner = {}
cudaStream_t stream = {}
bool speculativeRequest = {}
DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}
std::vector<int32_t> pendingCompactionMapping
int32_t pendingCompactionBatchSize = {-1}
Tensor const *pendingDeviceBatchMapping = {}
std::vector<SequenceState> sequences
std::vector<uint8_t> decodeParticipants

Execution indices that decode in the prepared step; empty means every sequence.

struct RequestSlotToken

Public Functions

inline explicit RequestSlotToken(
ContextCacheCoordinator &coordinator
) noexcept
inline RequestSlotToken(RequestSlotToken &&other) noexcept
RequestSlotToken &operator=(RequestSlotToken&&) = delete
RequestSlotToken(RequestSlotToken const&) = delete
RequestSlotToken &operator=(RequestSlotToken const&) = delete
inline ~RequestSlotToken() noexcept

Public Members

ContextCacheCoordinator *owner = {}
struct SequenceState

Public Members

CacheRequestLease lease
AdmissionState admissionState = {AdmissionState::kSettled}
bool admissionCheckpointRestored = {}
bool admissionChunkOpen = {}

A prepared prefill chunk whose device work the caller has not yet reported complete.

ResidentRef resident
std::vector<int32_t> tokenIds
BlockKeyExtras keyExtras
std::vector<Hash128> perPositionMediaHash
int32_t reuseTokenLength = {}
ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}
ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}
int32_t replayTailLength = {}
int32_t committedStateLength = {}
int32_t publishedFullBlockCount = {}
int32_t publishedExactLength = {}
int32_t commonStateLength = {}
std::optional<int32_t> frozenSpecPrefillLength
std::optional<StagedHybridPublication> stagedHybridPublication
struct StagedHybridPublication

Public Members

HybridCheckpointKey checkpoint
HybridSnapshotReservation snapshots
PublicationPoint point = {}
struct ContextCachePhysicalResources#

Validated physical resources borrowed from SharedResources for the coordinator lifetime.

Public Members

HybridCacheManager &baseCache#
KVPageTable &basePageTable#
HybridCacheManager *draftCache = {}#
KVPageTable *draftPageTable = {}#
KVPageTable *swaPageTable = {}#
struct ContextCacheSequenceAdmission#

One complete logical input admitted to the cache before its executable suffix is derived.

Public Members

std::vector<int32_t> tokenIds#
BlockKeyExtras keyExtras#

Request-wide non-token identity; LoRA/isolation identity is constant for the sequence.

std::vector<Hash128> perPositionMediaHash#

Per-position media content hash. Empty means text-only. When non-empty, must have tokenIds.size() entries. A non-zero Hash128 at position i causes the block hash to consume that 128-bit digest instead of the token ID.

ResidentRef resident#

Scheduler-owned physical state identity; the coordinator may not derive this from logical sequence order.

ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}#

Whether the sequence looks up and publishes reusable state. Bypass still uses managed private pages but neither looks up nor publishes.

ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}#

Which ready endpoints the sequence publishes.

int32_t replayTailLength = {0}#

Hybrid+spec replay tail length derived before cache admission.

struct ContextCacheBatchAdmission#

One serialized runtime request: the batch the cache steps together. Cache policies belong to the sequences; the request carries only what the deployment fixes for every sequence it will hold.

The sequences may be empty: such a request opens with no lease and takes every sequence through admitSequence() while it executes.

Public Members

std::vector<ContextCacheSequenceAdmission> sequences#
bool speculativeRequest = {}#
DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}#
struct ContextCacheSequenceAdvance#

Host-visible sequence advance observed after an existing stream synchronization.

Public Members

int32_t const *acceptedTokenIds = {}#
int32_t acceptedTokenCount = {}#
int32_t committedStateLength = {}#

Greatest logical token boundary whose model state is materialized in the bound cache.

Public Static Attributes

static constexpr int32_t kHoldCommittedStateLength = -1#

committedStateLength value for a zero advance that holds the ledger’s current committed length in place. A concrete length with no accepted tokens represents a state-only advance.

struct AdmissionResult

Public Members

RequestHandle request
std::vector<int32_t> prefillStarts

Per-sequence logical token offset at which runtime prefill begins.

std::vector<SwaKVCacheState> swaStates

Seeded states whose page references remain owned by the corresponding context-cache leases.

struct BeginRequestResult

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}
std::optional<AdmissionResult> admission
std::optional<int32_t> overlongPrefillLength = {}

Set on a refusal whose cause is a sequence’s uncached suffix exceeding maxPrefillLength(): that suffix length. Such a sequence is refused before anything is leased, so no record is evicted for it.

struct AdmitSequenceResult

Public Members

ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}
bool insufficientCapacity = {}

True when the refusal is transient pool pressure rather than an error: pages free as resident sequences retire, so the caller may retry at a later step boundary.

int32_t prefillStart = {}

Logical token offset at which the runtime’s prefill begins (== the reused prefix length); on an overlongPrefill refusal, the reused length the lookup found.

bool overlongPrefill = {}

Refused because the uncached suffix exceeds maxPrefillLength(); nothing was leased or evicted.