Context Cache Coordinator#
-
class ContextCacheCoordinator#
Owns the complete host/device lifecycle around the CUDA-free ContextCacheManager.
The coordinator has no worker of its own; every call is made by the caller’s thread and callers must keep those calls serialized. Request handles may overlap in lifetime: several may be admitted and hold a page lease at once, which is the ownership shape in-flight batching needs.
Overlapping leases are not overlapping execution, and the distinction is load-bearing. Page-table rows bind to stable physical resident slots, while request-local reuse-length staging is reused across preparations. A prepare issued before the previous request’s async copy has landed would corrupt that staging buffer. A caller must therefore still finish one request’s steps before preparing the next; what a count rather than a flag permits is that the leases, not the execution, may interleave.
Normal publication occurs only at host-visible completion points already present in the runtime. An abnormal exit drains the bound stream before releasing active page references; a failed drain quarantines the entire request and poisons the coordinator until runtime-owned shutdown can establish quiescence. Poisoning gates new admissions, so a request already admitted when the stream failed is quarantined by its own drain rather than by the first failure.
Sequence-admission transaction
Every sequence joined through admitSequence is in admission until finalizeSequenceAdmission commits its final prefill chunk or abortSequenceAdmission discards it. Any number of sequences may be in admission at once, at any execution index, and each tracks its own in-flight device work. Per chunk the caller runs prepare -> (final chunk only: capture) -> its own stream synchronization -> complete -> advance (non-final) or finalize (final).
- ContextCacheCoordinatorStatus prepareSequenceAdmission(
- RequestHandle &request,
- int32_t sequenceIndex
Open the next prefill chunk of a sequence in admission. Before the first chunk this restores the sequence’s recurrent and partial-KV checkpoint into its stable resident slot without touching any other resident sequence.
sequenceIndexis the transient execution index.
- ContextCacheCoordinatorStatus enqueueSequenceAdmissionCapture(
- RequestHandle &request,
- int32_t sequenceIndex
Reserve and enqueue the sequence’s prefill-end snapshot after its final chunk and before the runtime’s existing sampled-token synchronization. This is a no-op for non-hybrid deployments.
- ContextCacheCoordinatorStatus completeSequenceAdmissionDeviceWork(
- RequestHandle &request,
- int32_t sequenceIndex
Resolve the sequence’s open chunk after the runtime’s stream synchronization. Other sequences’ chunks stay open until their own completion.
- ContextCacheCoordinatorStatus advanceSequenceAdmission(
- RequestHandle &request,
- int32_t sequenceIndex,
- int32_t committedStateLength
Record a committed non-final prefill chunk: the slot’s state now covers the first
committedStateLengthprompt tokens. Nothing is published and no snapshot is captured.
- ContextCacheCoordinatorStatus abortSequenceAdmission(
- RequestHandle &request,
- int32_t sequenceIndex
Discard a failed or cancelled sequence-admission transaction so ordinary eviction can retire it; the eviction’s compaction releases its lease. Drains the stream only when the sequence has an open chunk. The sequence never publishes afterwards. Throws for a sequence that is not in admission (finalized, aborted, or admitted with the request).
- ContextCacheCoordinatorStatus finalizeSequenceAdmission(
- RequestHandle &request,
- int32_t sequenceIndex,
- ContextCacheSequenceAdvance const &advance,
- std::optional<int32_t> commonStateLength = std::nullopt
The single-slot form of finalizePrefillPublication: record the final chunk’s lookahead token, advance the slot’s committed prefix to the full input, and publish its ready prefix state (the hybrid snapshot captured for this sequence only).
commonStateLengthis the prefix a speculative sequence’s draft has materialized (the full input when empty). A Hybrid+spec sequence published its checkpoint during the prefill (publishHybridSpecEndpoint); an EAGLE-flavor sequence’s prompt endpoint publishes at the next decode completion, capped at the common state.
Public Types
-
using StreamSynchronizer = std::function<cudaError_t(cudaStream_t)>#
Public Functions
- ContextCacheCoordinator(
- ContextCacheConfig const &config,
- DeploymentConfig const &deployment,
- ContextCacheDeploymentProfile profile,
- ContextCachePhysicalResources resources,
- cudaStream_t stream,
- StreamSynchronizer synchronizer = {}
-
~ContextCacheCoordinator() noexcept#
-
ContextCacheCoordinator(ContextCacheCoordinator const&) = delete#
- ContextCacheCoordinator &operator=(
- ContextCacheCoordinator const&
- BeginRequestResult beginRequest(
- ContextCacheBatchAdmission const &admission,
- DecodingKvHeadroom const &headroom,
- cudaStream_t stream
- AdmitSequenceResult admitSequence(
- RequestHandle &request,
- ContextCacheSequenceAdmission const &admission,
- DecodingKvHeadroom const &headroom
Join one more sequence to a live request: look up its prefix, lease its pages, and bind exactly its page-table row (and its draft page-table row, for a paired speculative lease). Attention deployments without bounded SWA only.
The request may hold no sequence yet: an empty request admitted with no sequences takes its first one here, through the same path as every later one. The sequence inherits the request’s lookup, commit and replay-tail policies unless its admission overrides the lookup policy.
The device KV length for the new slot stays the runtime’s business – its seating sequence owns the cache-manager view – and the ledger is not advanced until advanceSequenceAdmission or finalizeSequenceAdmission reports a committed prefill chunk.
Other sequences may still be in admission, and their chunk work may be in flight: the lease comes from pages no enqueued work references, and the row upload is stream-ordered for a slot no enqueued step reads. Request-wide work (a decode step or a compaction) must be resolved first.
-
bool retractSequenceAdmission(RequestHandle &request) noexcept#
Undo the most recent admitSequence before its slot ever joined the runtime batch: release the lease, drop the sequence, and clear its page-table row.
Only legal while the admitted sequence is still the batch’s tail and the runtime holds no slot for it – the recovery path for a seating that threw between lease and slot append. A seated slot that failed later stays, terminal from birth, and leaves through the ordinary eviction instead. Retracting from a request that holds no sequence poisons the coordinator: nothing was admitted, so the caller’s bookkeeping is already wrong.
-
ContextCacheCoordinatorStatus preparePrefill(RequestHandle &request)#
Bind every admitted row and reset logical cache lengths to the selected reuse boundaries, entering execution. A request admitted with no sequences has no prefill to complete and is settled at once.
- ContextCacheCoordinatorStatus enqueuePrefillCaptures(
- RequestHandle &request
Reserve and enqueue hybrid prefill-end snapshots before the runtime’s existing prefill synchronization.
- ContextCacheCoordinatorStatus finalizePrefillPublication(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths = nullptr,
- SwaWindowStateView const *swaWindowStates = nullptr
Apply the post-prefill sequence advance and publish every ready full-block endpoint. For EAGLE, commonStateLengths caps publication at the prefix materialized by both base and draft state.
- ContextCacheCoordinatorStatus publishHybridSpecEndpoint(
- RequestHandle &request,
- int32_t slot,
- int32_t residentStateLength,
- Tensor const &baseHiddenStates,
- int32_t boundaryHiddenRow
Publish one Hybrid+spec checkpoint at the stable predecessor boundary. This is the dedicated MTP publication entrypoint; the runtime drives it after the folded draft prefill has materialized the boundary draft state, for a blocking request’s sequence or one in sequence admission. It captures the recurrent state, the paired base+draft partial pages, and the successor-dependent boundary base-hidden row, then commits the exact checkpoint. Skipped for bypass, already-published, or empty prefixes.
- ContextCacheCoordinatorStatus restoreHybridSpecBoundaryHidden(
- RequestHandle &request,
- int32_t slot,
- Tensor &baseHiddenStates,
- int32_t destinationRow
Restore the checkpoint’s saved boundary base-hidden row into baseHiddenStates[slot, destinationRow, :] for the runtime’s fold micro-forward. No synchronization: the caller orders this within its prefill stream.
- ContextCacheCoordinatorStatus prepareDecodeStep(
- RequestHandle &request,
- DecodingKvHeadroom const &headroom,
- std::vector<int32_t> const *extraTokensPerSlot = nullptr,
- std::vector<int32_t> const *draftExtraTokensPerSlot = nullptr
Grow and upload every row needed for the next decode working set before model execution.
extraTokensPerSlot, when set, is each sequence’s own growth beyond its committed state (one, or the pending run a jump-forward left; a jump-forward slot’s wider verify round) and replaces headroom.baseExtraTokens.draftExtraTokensPerSlotlikewise replaces headroom.draftExtraTokens; paired speculative requests only.
- ContextCacheCoordinatorStatus prepareDecodeStep(
- RequestHandle &request,
- DecodingKvHeadroom const &headroom,
- std::vector<int32_t> const &decodeSequences
Grow only the rows of
decodeSequences(execution indices, each with a complete prompt) for a step in which the other sequences do not decode. Every other sequence must carry the hold advance in the matching completeDecodeStep. Non-speculative requests only.
- ContextCacheCoordinatorStatus completeDecodeStep(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths = nullptr,
- SwaWindowStateView const *swaWindowStates = nullptr
Apply the post-decode sequence advance after the decoder’s existing synchronization. A sequence that did not decode this step – outside the prepared decode subset, or still in admission – must carry the hold advance and cannot be published. For EAGLE, commonStateLengths excludes any unmaterialized accepted suffix and speculative lookahead.
- bool isExternallyOwnedSwaPage(
- RequestHandle const &request,
- int32_t slot,
- PageId page
- PageId replaceSwaPage(
- RequestHandle &request,
- int32_t slot,
- PageId page
- ContextCacheCoordinatorStatus beginBatchCompaction(
- RequestHandle &request,
- std::vector<int32_t> const &oldToNew,
- int32_t newBatchSize,
- Tensor &deviceBatchMapping
Validate and upload the one authoritative old-to-new mapping before any old-slot retirement work.
-
ContextCacheCoordinatorStatus compactBatch(RequestHandle &request)#
Retire logical sequences while preserving survivor physical rows, then consume the eviction sync.
-
ContextCacheCoordinatorStatus finish(RequestHandle &request)#
Consume a normally completed request. This is idempotent for an already-empty handle.
-
ContextCacheCoordinatorStatus shutdown() noexcept#
Drain quarantined ownership before physical cache resources are destroyed.
-
ContextCacheMetrics metrics() const noexcept#
-
ContextCacheManager const &manager() const noexcept#
-
ContextCacheManager &manager() noexcept#
-
bool supportsLiveSequenceAdmission() const noexcept#
Whether this deployment can lease and bind a new sequence into an executing request.
-
int32_t maxPrefillLength() const noexcept#
The longest uncached suffix an admitted sequence may leave to prefill: the engine’s maximum input length.
-
int32_t activeRequests() const noexcept#
Requests begun and not yet finished or abandoned, quarantined ones included.
-
struct AcquireSequenceResult#
-
struct AdmissionResult#
Public Members
-
RequestHandle request#
-
std::vector<int32_t> prefillStarts#
Per-sequence logical token offset at which runtime prefill begins.
-
std::vector<SwaKVCacheState> swaStates#
Seeded states whose page references remain owned by the corresponding context-cache leases.
-
RequestHandle request#
-
struct AdmitSequenceResult#
Public Members
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#
-
bool insufficientCapacity = {}#
True when the refusal is transient pool pressure rather than an error: pages free as resident sequences retire, so the caller may retry at a later step boundary.
-
int32_t prefillStart = {}#
Logical token offset at which the runtime’s prefill begins (== the reused prefix length); on an overlongPrefill refusal, the reused length the lookup found.
-
bool overlongPrefill = {}#
Refused because the uncached suffix exceeds maxPrefillLength(); nothing was leased or evicted.
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#
-
class BaseEndpointPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#
Public Functions
-
inline virtual char const *name() const noexcept override#
- inline virtual void onPrefillFinalized(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.
- inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-decode advance and publish endpoints for the completed slots.
- inline explicit PublicationPolicy(
- ContextCacheCoordinator &coordinator
-
inline virtual char const *name() const noexcept override#
-
struct BeginRequestResult#
Public Members
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#
-
std::optional<AdmissionResult> admission#
-
std::optional<int32_t> overlongPrefillLength = {}#
Set on a refusal whose cause is a sequence’s uncached suffix exceeding maxPrefillLength(): that suffix length. Such a sequence is refused before anything is leased, so no record is evicted for it.
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}#
-
class EagleSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#
Public Functions
-
inline virtual char const *name() const noexcept override#
- inline virtual void onPrefillFinalized(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const*
Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.
- inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const*
Apply the post-decode advance and publish endpoints for the completed slots.
- inline virtual void onSequenceAdmissionFinalized(
- RequestHandle &request,
- int32_t sequenceIndex
A sequence admitted into the executing request seated its draft prefill without a synchronization: its prompt endpoint freezes now and publishes, capped at the common state, at the next decode completion (whose verification synchronizes) or at terminalization.
- inline virtual ContextCacheCoordinatorStatus onTerminalize(
- RequestHandle &request
EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.
- inline virtual ContextCacheCoordinatorStatus onCompaction(
- RequestHandle &request,
- std::vector<int32_t> const &oldToNew
A seated sequence that survives the compaction keeps its frozen endpoint for its first decode completion; only a retiring one publishes now.
- inline explicit PublicationPolicy(
- ContextCacheCoordinator &coordinator
-
inline virtual char const *name() const noexcept override#
-
class HybridSnapshotPolicy : public trt_edgellm::rt::ContextCacheCoordinator::PublicationPolicy#
Subclassed by trt_edgellm::rt::ContextCacheCoordinator::HybridSpecPolicy
Public Functions
-
inline virtual char const *name() const noexcept override#
- inline virtual void onPrefillFinalized(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const*
Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.
- inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const*
Apply the post-decode advance and publish endpoints for the completed slots.
- inline virtual void onSequenceAdmissionFinalized(
- RequestHandle &request,
- int32_t sequenceIndex
A sequence admitted into the executing request committed its prompt: publish (or arm) its prefill-end endpoint. The default publishes the ready full-block base endpoint.
- inline explicit PublicationPolicy(
- ContextCacheCoordinator &coordinator
-
inline virtual char const *name() const noexcept override#
-
class HybridSpecPolicy : public trt_edgellm::rt::ContextCacheCoordinator::HybridSnapshotPolicy#
Public Functions
-
inline virtual char const *name() const noexcept override#
- inline virtual void onPrefillFinalized(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const*
Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.
- inline virtual ContextCacheCoordinatorStatus onDecodeCompleted(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-decode advance and publish endpoints for the completed slots.
- inline virtual void onSequenceAdmissionFinalized(
- RequestHandle &request,
- int32_t sequenceIndex
The checkpoint was published during the prefill (publishMtpBoundary), once the draft had seated the prompt.
- inline virtual ContextCacheCoordinatorStatus publishMtpBoundary(
- RequestHandle &request,
- int32_t slot,
- int32_t residentStateLength,
- Tensor const &baseHiddenStates,
- int32_t boundaryHiddenRow
Hybrid+spec-only: publish the successor-boundary checkpoint. Unreachable for other flavors.
- inline virtual ContextCacheCoordinatorStatus restoreMtpBoundary(
- RequestHandle &request,
- int32_t slot,
- Tensor &baseHiddenStates,
- int32_t destinationRow
Hybrid+spec-only: restore the checkpoint’s saved boundary hidden row. Unreachable for other flavors.
-
inline virtual char const *name() const noexcept override#
-
class PublicationPolicy#
Subclassed by trt_edgellm::rt::ContextCacheCoordinator::BaseEndpointPolicy, trt_edgellm::rt::ContextCacheCoordinator::EagleSpecPolicy, trt_edgellm::rt::ContextCacheCoordinator::HybridSnapshotPolicy, trt_edgellm::rt::ContextCacheCoordinator::SharedKvSpecPolicy
Public Functions
- inline explicit PublicationPolicy(
- ContextCacheCoordinator &coordinator
-
virtual ~PublicationPolicy() noexcept = default#
-
virtual char const *name() const noexcept = 0#
- virtual void onPrefillFinalized(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.
- virtual ContextCacheCoordinatorStatus onDecodeCompleted(
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-decode advance and publish endpoints for the completed slots.
- inline virtual void onSequenceAdmissionFinalized(
- RequestHandle &request,
- int32_t sequenceIndex
A sequence admitted into the executing request committed its prompt: publish (or arm) its prefill-end endpoint. The default publishes the ready full-block base endpoint.
- inline virtual ContextCacheCoordinatorStatus onTerminalize( )#
EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.
- inline virtual ContextCacheCoordinatorStatus onCompaction(
- RequestHandle &request,
- std::vector<int32_t> const&
Before a compaction that retires the rows
oldToNewmaps below zero. Default: terminalize.
- inline virtual ContextCacheCoordinatorStatus publishMtpBoundary(
- RequestHandle&,
- int32_t,
- int32_t,
- Tensor const&,
- int32_t
Hybrid+spec-only: publish the successor-boundary checkpoint. Unreachable for other flavors.
- inline virtual ContextCacheCoordinatorStatus restoreMtpBoundary(
- RequestHandle&,
- int32_t,
- Tensor&,
- int32_t
Hybrid+spec-only: restore the checkpoint’s saved boundary hidden row. Unreachable for other flavors.
-
class RequestHandle#
Public Functions
-
RequestHandle(RequestHandle &&other) noexcept#
-
RequestHandle &operator=(RequestHandle &&other) = delete#
-
RequestHandle(RequestHandle const&) = delete#
-
RequestHandle &operator=(RequestHandle const&) = delete#
-
~RequestHandle() noexcept#
-
bool valid() const noexcept#
-
struct Impl#
Public Types
Public Functions
- inline explicit Impl(
- ContextCacheCoordinator &coordinator,
- cudaStream_t requestStream
-
inline bool admitted() const noexcept#
-
inline bool executing() const noexcept#
-
inline bool hasPendingDeviceWork() const noexcept#
-
inline bool hasPendingRequestWork() const noexcept#
-
inline bool hasUnsynchronizedSeatedPrefill() const noexcept#
A sequence seated into the executing request froze its prompt endpoint over a draft prefill that nothing has synchronized yet: it is synchronized by the next decode verification or at terminalization.
-
inline bool hasOpenAdmissionChunk() const noexcept#
-
inline bool decodesThisStep(size_t sequenceIndex) const noexcept#
- inline SequenceState &sequenceInAdmission(
- int32_t sequenceIndex,
- char const *operation
-
inline bool awaitingFirstSpecCompletion() const noexcept#
-
inline void beginPrefill() noexcept#
kAdmitted -> kExecuting with the prefill work enqueued on the stream.
-
inline void markDeviceWorkEnqueued() noexcept#
Fresh GPU work was enqueued on impl.stream (prefill grow, decode grow, fold, compaction copy).
-
inline void markDeviceWorkSynchronized() noexcept#
impl.stream was explicitly synchronized (or a verification round already synchronized it).
-
inline void markDeviceWorkResolvedWithoutSync() noexcept#
The enqueued work is provably terminal without a coordinator synchronize (no async publish work remains).
-
inline void beginSpecDraftInit() noexcept#
EAGLE only: the ordered draft initialization awaits its first verification round before publishing.
-
inline void endSpecDraftInit() noexcept#
Public Members
-
RequestSlotToken requestSlot#
Declared before sequences so its destructor drops the active-request count only after every lease is released.
-
std::unique_ptr<PublicationPolicy> publicationPolicy#
-
ContextCacheCoordinator *owner = {}#
-
cudaStream_t stream = {}#
-
bool speculativeRequest = {}#
-
DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}#
-
std::vector<int32_t> pendingCompactionMapping#
-
int32_t pendingCompactionBatchSize = {-1}#
-
std::vector<SequenceState> sequences#
-
std::vector<uint8_t> decodeParticipants#
Execution indices that decode in the prepared step; empty means every sequence.
-
struct RequestSlotToken#
Public Functions
- inline explicit RequestSlotToken(
- ContextCacheCoordinator &coordinator
-
inline RequestSlotToken(RequestSlotToken &&other) noexcept#
-
RequestSlotToken &operator=(RequestSlotToken&&) = delete#
-
RequestSlotToken(RequestSlotToken const&) = delete#
-
RequestSlotToken &operator=(RequestSlotToken const&) = delete#
-
inline ~RequestSlotToken() noexcept#
Public Members
-
ContextCacheCoordinator *owner = {}#
-
struct SequenceState#
Public Members
-
CacheRequestLease lease#
-
AdmissionState admissionState = {AdmissionState::kSettled}#
-
bool admissionCheckpointRestored = {}#
-
bool admissionChunkOpen = {}#
A prepared prefill chunk whose device work the caller has not yet reported complete.
-
ResidentRef resident#
-
std::vector<int32_t> tokenIds#
-
BlockKeyExtras keyExtras#
-
int32_t reuseTokenLength = {}#
-
ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}#
-
ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}#
-
int32_t replayTailLength = {}#
-
int32_t committedStateLength = {}#
-
int32_t publishedFullBlockCount = {}#
-
int32_t publishedExactLength = {}#
-
int32_t commonStateLength = {}#
-
std::optional<int32_t> frozenSpecPrefillLength#
-
std::optional<StagedHybridPublication> stagedHybridPublication#
-
CacheRequestLease lease#
-
struct StagedHybridPublication#
Public Members
-
HybridCheckpointKey checkpoint#
-
HybridSnapshotReservation snapshots#
-
PublicationPoint point = {}#
-
HybridCheckpointKey checkpoint#
-
RequestHandle(RequestHandle &&other) noexcept#
Public Functions
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-prefill advance and publish (or arm) every ready endpoint for this flavor.
- RequestHandle &request,
- std::vector<ContextCacheSequenceAdvance> const &advances,
- std::vector<int32_t> const &publishableCompletedSlots,
- std::vector<int32_t> const *commonStateLengths,
- SwaWindowStateView const *swaWindowStates
Apply the post-decode advance and publish endpoints for the completed slots.
- RequestHandle &request,
- int32_t sequenceIndex
A sequence admitted into the executing request committed its prompt: publish (or arm) its prefill-end endpoint. The default publishes the ready full-block base endpoint.
- RequestHandle &request
EAGLE-only: publish the frozen prefill once the first verification round terminalizes. Default no-op.
- RequestHandle &request,
- std::vector<int32_t> const &oldToNew
Before a compaction that retires the rows
oldToNewmaps below zero. Default: terminalize.
- ContextCacheCoordinator &coordinator
-
class RequestHandle
Public Functions
-
RequestHandle(RequestHandle &&other) noexcept
-
RequestHandle &operator=(RequestHandle &&other) = delete
-
RequestHandle(RequestHandle const&) = delete
-
RequestHandle &operator=(RequestHandle const&) = delete
-
~RequestHandle() noexcept
-
bool valid() const noexcept
-
struct Impl
Public Types
-
enum class Phase : uint8_t
Values:
-
enumerator kAdmitted
-
enumerator kExecuting
-
enumerator kFinishing
-
enumerator kAdmitted
-
enum class AdmissionState : uint8_t
Sequences admitted with the request start settled. A live admission stays kAdmitting until its final prefill chunk commits; an aborted one never publishes and waits for eviction.
Values:
-
enumerator kSettled
-
enumerator kAdmitting
-
enumerator kAborted
-
enumerator kSettled
Public Functions
- inline explicit Impl(
- ContextCacheCoordinator &coordinator,
- cudaStream_t requestStream
-
inline bool admitted() const noexcept
-
inline bool executing() const noexcept
-
inline bool hasPendingDeviceWork() const noexcept
-
inline bool hasPendingRequestWork() const noexcept
-
inline bool hasUnsynchronizedSeatedPrefill() const noexcept
A sequence seated into the executing request froze its prompt endpoint over a draft prefill that nothing has synchronized yet: it is synchronized by the next decode verification or at terminalization.
-
inline bool hasOpenAdmissionChunk() const noexcept
-
inline bool decodesThisStep(size_t sequenceIndex) const noexcept
- inline SequenceState &sequenceInAdmission(
- int32_t sequenceIndex,
- char const *operation
-
inline bool awaitingFirstSpecCompletion() const noexcept
-
inline void beginPrefill() noexcept
kAdmitted -> kExecuting with the prefill work enqueued on the stream.
-
inline void markDeviceWorkEnqueued() noexcept
Fresh GPU work was enqueued on impl.stream (prefill grow, decode grow, fold, compaction copy).
-
inline void markDeviceWorkSynchronized() noexcept
impl.stream was explicitly synchronized (or a verification round already synchronized it).
-
inline void markDeviceWorkResolvedWithoutSync() noexcept
The enqueued work is provably terminal without a coordinator synchronize (no async publish work remains).
-
inline void beginSpecDraftInit() noexcept
EAGLE only: the ordered draft initialization awaits its first verification round before publishing.
-
inline void endSpecDraftInit() noexcept
-
inline void markFinishing() noexcept
Enter the terminal phase after an unrecoverable capacity/growth failure. Normal completion is the caller’s finish(); a batch that drained to nothing stays executing until then.
Public Members
-
RequestSlotToken requestSlot
Declared before sequences so its destructor drops the active-request count only after every lease is released.
-
std::unique_ptr<PublicationPolicy> publicationPolicy
-
ContextCacheCoordinator *owner = {}
-
cudaStream_t stream = {}
-
bool speculativeRequest = {}
-
DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}
-
std::vector<int32_t> pendingCompactionMapping
-
int32_t pendingCompactionBatchSize = {-1}
-
Tensor const *pendingDeviceBatchMapping = {}
-
std::vector<SequenceState> sequences
-
std::vector<uint8_t> decodeParticipants
Execution indices that decode in the prepared step; empty means every sequence.
-
struct RequestSlotToken
Public Functions
- inline explicit RequestSlotToken(
- ContextCacheCoordinator &coordinator
-
inline RequestSlotToken(RequestSlotToken &&other) noexcept
-
RequestSlotToken &operator=(RequestSlotToken&&) = delete
-
RequestSlotToken(RequestSlotToken const&) = delete
-
RequestSlotToken &operator=(RequestSlotToken const&) = delete
-
inline ~RequestSlotToken() noexcept
Public Members
-
ContextCacheCoordinator *owner = {}
-
struct SequenceState
Public Members
-
CacheRequestLease lease
-
AdmissionState admissionState = {AdmissionState::kSettled}
-
bool admissionCheckpointRestored = {}
-
bool admissionChunkOpen = {}
A prepared prefill chunk whose device work the caller has not yet reported complete.
-
ResidentRef resident
-
std::vector<int32_t> tokenIds
-
BlockKeyExtras keyExtras
-
std::vector<Hash128> perPositionMediaHash
-
int32_t reuseTokenLength = {}
-
ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}
-
ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}
-
int32_t replayTailLength = {}
-
int32_t committedStateLength = {}
-
int32_t publishedFullBlockCount = {}
-
int32_t publishedExactLength = {}
-
int32_t commonStateLength = {}
-
std::optional<int32_t> frozenSpecPrefillLength
-
std::optional<StagedHybridPublication> stagedHybridPublication
-
CacheRequestLease lease
-
struct StagedHybridPublication
Public Members
-
HybridCheckpointKey checkpoint
-
HybridSnapshotReservation snapshots
-
PublicationPoint point = {}
-
HybridCheckpointKey checkpoint
-
enum class Phase : uint8_t
-
RequestHandle(RequestHandle &&other) noexcept
-
struct ContextCachePhysicalResources#
Validated physical resources borrowed from SharedResources for the coordinator lifetime.
Public Members
-
HybridCacheManager &baseCache#
-
KVPageTable &basePageTable#
-
HybridCacheManager *draftCache = {}#
-
KVPageTable *draftPageTable = {}#
-
KVPageTable *swaPageTable = {}#
-
HybridCacheManager &baseCache#
-
struct ContextCacheSequenceAdmission#
One complete logical input admitted to the cache before its executable suffix is derived.
Public Members
-
std::vector<int32_t> tokenIds#
-
BlockKeyExtras keyExtras#
Request-wide non-token identity; LoRA/isolation identity is constant for the sequence.
-
std::vector<Hash128> perPositionMediaHash#
Per-position media content hash. Empty means text-only. When non-empty, must have tokenIds.size() entries. A non-zero Hash128 at position i causes the block hash to consume that 128-bit digest instead of the token ID.
-
ResidentRef resident#
Scheduler-owned physical state identity; the coordinator may not derive this from logical sequence order.
-
ContextCacheLookupPolicy lookupPolicy = {ContextCacheLookupPolicy::kUseCache}#
Whether the sequence looks up and publishes reusable state. Bypass still uses managed private pages but neither looks up nor publishes.
-
ContextCacheCommitPolicy commitPolicy = {ContextCacheCommitPolicy::kIncludingGeneratedTokens}#
Which ready endpoints the sequence publishes.
-
int32_t replayTailLength = {0}#
Hybrid+spec replay tail length derived before cache admission.
-
std::vector<int32_t> tokenIds#
-
struct ContextCacheBatchAdmission#
One serialized runtime request: the batch the cache steps together. Cache policies belong to the sequences; the request carries only what the deployment fixes for every sequence it will hold.
The sequences may be empty: such a request opens with no lease and takes every sequence through admitSequence() while it executes.
Public Members
-
std::vector<ContextCacheSequenceAdmission> sequences#
-
bool speculativeRequest = {}#
-
DecodingTokenStateContract tokenStateContract = {DecodingTokenStateContract::kCommittedPlusLookahead}#
-
std::vector<ContextCacheSequenceAdmission> sequences#
-
struct ContextCacheSequenceAdvance#
Host-visible sequence advance observed after an existing stream synchronization.
Public Members
-
int32_t const *acceptedTokenIds = {}#
-
int32_t acceptedTokenCount = {}#
-
int32_t committedStateLength = {}#
Greatest logical token boundary whose model state is materialized in the bound cache.
Public Static Attributes
-
static constexpr int32_t kHoldCommittedStateLength = -1#
committedStateLength value for a zero advance that holds the ledger’s current committed length in place. A concrete length with no accepted tokens represents a state-only advance.
-
int32_t const *acceptedTokenIds = {}#
-
struct AdmissionResult
Public Members
-
RequestHandle request
-
std::vector<int32_t> prefillStarts
Per-sequence logical token offset at which runtime prefill begins.
-
std::vector<SwaKVCacheState> swaStates
Seeded states whose page references remain owned by the corresponding context-cache leases.
-
RequestHandle request
-
struct BeginRequestResult
Public Members
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}
-
std::optional<AdmissionResult> admission
-
std::optional<int32_t> overlongPrefillLength = {}
Set on a refusal whose cause is a sequence’s uncached suffix exceeding maxPrefillLength(): that suffix length. Such a sequence is refused before anything is leased, so no record is evicted for it.
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}
-
struct AdmitSequenceResult
Public Members
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}
-
bool insufficientCapacity = {}
True when the refusal is transient pool pressure rather than an error: pages free as resident sequences retire, so the caller may retry at a later step boundary.
-
int32_t prefillStart = {}
Logical token offset at which the runtime’s prefill begins (== the reused prefix length); on an overlongPrefill refusal, the reused length the lookup found.
-
bool overlongPrefill = {}
Refused because the uncached suffix exceeds maxPrefillLength(); nothing was leased or evicted.
-
ContextCacheCoordinatorStatus status = {ContextCacheCoordinatorStatus::kRequestFailed}