Hybrid Cache Manager#
-
class HybridCacheManager#
Top-level cache manager for hybrid Attention + Mamba architectures.
Routes cache access by absolute decoder-layer index to the appropriate sub-manager (
KVCacheManagerfor attention layers,MambaCacheManagerfor recurrent layers). Owns the shared device KV cache lengths tensor and provides unified compaction, prompt cache capture / restore, and batch management APIs that span both sub-managers.Public Types
Public Functions
-
HybridCacheManager() noexcept = default#
Default constructor (no allocation)
-
HybridCacheManager(Config const &config, cudaStream_t stream)#
Construct and initialise sub-managers and routing tables.
Builds absolute-to-local index mapping for each sub-manager, constructs
KVCacheManagerandMambaCacheManager, and allocates the shared device KV-cache-lengths tensor (zero-initialised).- Parameters:
config – Cache configuration with per-layer type routing
stream – CUDA stream for allocation and memset
-
~HybridCacheManager() noexcept#
Destructor.
-
HybridCacheManager(HybridCacheManager const&) = delete#
Deleted copy constructor.
-
HybridCacheManager &operator=(HybridCacheManager const&) = delete#
Deleted copy assignment.
- Returns:
Reference to this
-
HybridCacheManager(HybridCacheManager&&) noexcept#
Move constructor.
-
HybridCacheManager &operator=(HybridCacheManager&&) noexcept#
Move assignment operator.
- Returns:
Reference to this
-
inline int32_t numLayers() const noexcept#
Total number of decoder layers (attention + Mamba).
- Returns:
Number of layers (== size of the configured layer-type list).
-
LayerType getLayerType(int32_t absLayerIdx) const#
Type (attention or Mamba/recurrent) of a given absolute layer index.
- Parameters:
absLayerIdx – Absolute decoder-layer index in [0, numLayers()).
- Returns:
The layer’s type.
-
rt::Tensor &getCombinedKVCache(int32_t absLayerIdx)#
Get the combined KV-cache page pool for a given absolute layer index (must be an attention layer).
- Parameters:
absLayerIdx – Absolute decoder-layer index.
- Returns:
Reference to the page pool [2, numPages, kTOKENS_PER_PAGE, numKVHeads, headDim].
- std::pair<rt::Tensor, rt::Tensor> getSeparateKVCache(
- int32_t absLayerIdx
Get the K-half and V-half of a given attention layer’s pool as separate tensor views.
- Parameters:
absLayerIdx – Absolute decoder-layer index.
- Returns:
{kView, vView}, each shaped [maxBatch, capPadded, numKVHeads, headDim].
- KVLayerStorageMetadata getKVLayerStorageMetadata(
- int32_t absLayerIdx
Describe the physical pool and logical sequence geometry used by an attention layer.
-
rt::Tensor &getRecurrentState(int32_t absLayerIdx)#
Get the recurrent state for a given absolute layer index (must be a Mamba layer).
- Parameters:
absLayerIdx – Absolute decoder-layer index.
- Returns:
Reference to the per-layer recurrent state tensor.
-
rt::Tensor &getConvState(int32_t absLayerIdx, int32_t stateIdx = 0)#
Get a conv state for a given absolute layer index.
- Parameters:
absLayerIdx – Absolute decoder-layer index.
stateIdx – 0 for the layer’s own mixer state, which requires a Mamba layer; above that, one of the configured auxiliary states on that layer.
- Returns:
Reference to the per-slot conv state tensor.
-
int32_t numAuxConvStates() const noexcept#
Number of configured auxiliary conv states, across all layers.
- Returns:
Count, zero for a model that declares none.
-
AuxConvStateConfig const &auxConvStateConfig(int32_t auxIdx) const#
The configuration one auxiliary conv state was allocated from.
- Parameters:
auxIdx – Index in [0, numAuxConvStates()).
- Returns:
The entry, whose absLayerIdx names the layer it belongs to.
-
rt::Tensor &getAuxConvState(int32_t auxIdx)#
Storage for one auxiliary conv state, shaped [maxBatchSize, convDim, convWidth].
- Parameters:
auxIdx – Index in [0, numAuxConvStates()).
- Returns:
Reference to the owned tensor.
-
bool hasIntermediateAuxConvStates() const noexcept#
Whether the auxiliary conv states have spec-verify snapshots.
- Returns:
True when allocated, which follows the Mamba intermediate states.
-
rt::Tensor &getIntermediateAuxConvState(int32_t auxIdx)#
Per-token spec-verify snapshot of one auxiliary conv state.
- Parameters:
auxIdx – Index in [0, numAuxConvStates()).
- Returns:
Reference to the owned [maxBatchSize, maxIntermediateSeqLen, convDim, convWidth] tensor.
- void reshapeIntermediateAuxConvStates(
- int32_t activeBatchSize,
- int32_t seqLen
Reshape the auxiliary snapshots to a step’s extents.
- Parameters:
activeBatchSize – Sequences in the step.
seqLen – Verify tokens per sequence.
- void scatterAcceptedAuxConvStates(
- rt::Tensor const &acceptLengths,
- rt::Tensor const &stateIndices,
- cudaStream_t stream,
- rt::Tensor const *acceptedStateNodeIds = nullptr
Copy each sequence’s last accepted snapshot into its resident slot.
- Parameters:
acceptLengths – [activeBatchSize] INT32 accepted token count per sequence.
stateIndices – [activeBatchSize] INT32 resident slot per sequence.
stream – CUDA stream.
acceptedStateNodeIds – Tree verify: [activeBatchSize, maxAcceptLen] INT32 accepted node ids in path order. Null for chain verify, where the snapshot is
acceptLengths[b] - 1.
-
KVCacheManager &getKVCacheManager() noexcept#
Direct access to the KV cache sub-manager.
- Returns:
Reference to the KVCacheManager.
-
KVCacheManager const &getKVCacheManager() const noexcept#
-
MambaCacheManager &getMambaCacheManager() noexcept#
Direct access to the Mamba state sub-manager.
- Returns:
Reference to the MambaCacheManager.
-
std::vector<KVHeadDimGroupView> getKVHeadDimGroups() const#
Read-only views of the pre-computed KV head-dim groups.
Uniform models return a single group; hybrid Gemma4-style models return one group per distinct head dim. The underlying
KVLayerInfoarrays are owned by this manager and remain valid for its lifetime.- Returns:
Vector of group views (one per distinct head dim).
-
rt::Tensor &getKVCacheLengths() noexcept#
Get the shared device KV cache lengths tensor.
- Returns:
Reference to the device tensor of shape [activeBatchSize].
- void resetForNewSequences(
- rt::Tensor const &reuseKVCacheLengths,
- cudaStream_t stream
Reset state for new sequences. Validates batch size, copies reuse lengths from host to device, and updates the “all empty” flag.
- Parameters:
reuseKVCacheLengths – Host INT32 tensor with reuse lengths, shape [batchSize].
stream – CUDA stream.
- void commitSequenceLength(
- rt::Tensor const &newContextLengths,
- cudaStream_t stream
Commit sequence lengths after prefill (element-wise increment from a GPU tensor).
- Parameters:
newContextLengths – GPU INT32 tensor of context lengths, shape [activeBatchSize].
stream – CUDA stream.
-
void commitSequenceLength(int32_t increment, cudaStream_t stream)#
Commit sequence lengths after decode (scalar increment for all active sequences).
- Parameters:
increment – Scalar increment (typically 1).
stream – CUDA stream.
-
int32_t getActiveBatchSize() const noexcept#
Get the number of active sequences.
- Returns:
Active batch size.
-
void setActiveBatchSize(int32_t newActiveBatchSize)#
Set active batch size (used after batch eviction).
- Parameters:
newActiveBatchSize – New active batch size.
- Throws:
std::runtime_error – if out of range [0, maxBatchSize].
-
void restoreActiveBatchSize(int32_t activeBatchSize) noexcept#
Restore a previously validated batch-size view from an exception guard.
- void swapExecutionRows(
- int32_t rowA,
- int32_t rowB,
- cudaStream_t stream
Exchange the KV lengths of two execution rows. Self-inverse; the device counterpart of DecodingInferenceContext::swapExecutionRows, which a batch-one projection pairs it with so the projected row runs as row zero while every other row’s length is kept where it belongs.
- Throws:
std::runtime_error – if either row is outside the active batch or the swap cannot be enqueued.
-
void appendSequence(int32_t committedLength, cudaStream_t stream)#
Append one sequence to the active batch, as the last execution row, with the KV length it already holds (
committedLength:a reused prefix, or zero). The runtime session reserves the row with the sequence’s other reservations so the batch here follows the context’s from the seat on, prefilled or not. Strong exception guarantee: a throw leaves the batch as it was.- Throws:
std::runtime_error – when the batch is full or the device update cannot be enqueued.
- void materializeExecutionLengths(
- rt::Tensor const &executionLengths,
- cudaStream_t stream
Materialize the logical lengths for the next contiguous execution batch.
The source is indexed by execution row. This updates only the transient length input read by the engines; KV pages and recurrent state remain indexed by their resident slots.
-
bool getKVCacheAllEmpty() const noexcept#
Check if KV cache for all sequences is empty.
- Returns:
True if no prefill has been committed yet.
- void compactKVCacheLengths(
- rt::Tensor const &batchMapping,
- int32_t oldBatch,
- int32_t newBatch,
- cudaStream_t stream
Compact execution-row-aligned KV lengths without moving resident slot state.
-
void clearResidentSlot(int32_t slot, cudaStream_t stream)#
Clear one retired resident row’s recurrent/convolution state without moving survivors.
- std::vector<rt::Tensor> captureKVCache(
- int32_t batchIdx,
- int32_t sequenceLength,
- cudaStream_t stream
Capture KV cache for a single batch slot across all attention layers.
- Parameters:
batchIdx – Batch slot to capture.
sequenceLength – Number of tokens to capture from the cache.
stream – CUDA stream.
- Returns:
Vector of captured tensors (one per attention layer).
- void restoreKVCache(
- std::vector<rt::Tensor> const &saved,
- int32_t batchIdx,
- cudaStream_t stream
Restore KV cache for a single batch slot across all attention layers.
- Parameters:
saved – Previously captured KV cache tensors.
batchIdx – Target batch slot.
stream – CUDA stream.
- std::vector<rt::Tensor> captureRecurrentStates(
- int32_t batchIdx,
- cudaStream_t stream
Capture recurrent states for a single batch slot (delegates to MambaCacheManager).
- Parameters:
batchIdx – Batch slot to capture.
stream – CUDA stream.
- Returns:
Vector of captured tensors (one per recurrent layer).
- std::vector<rt::Tensor> captureConvStates(
- int32_t batchIdx,
- cudaStream_t stream
Capture conv states for a single batch slot (delegates to MambaCacheManager).
- Parameters:
batchIdx – Batch slot to capture.
stream – CUDA stream.
- Returns:
Vector of captured tensors (one per recurrent layer).
- std::vector<rt::Tensor> captureAuxConvStates(
- int32_t slot,
- cudaStream_t stream
Capture every auxiliary conv state for a single resident slot.
- Parameters:
slot – Resident state-pool row to capture, as in
state_indices.stream – CUDA stream.
- Returns:
One tensor per auxiliary state, in configured order; empty when there are none.
- void restoreAuxConvStates(
- std::vector<rt::Tensor> const &saved,
- int32_t slot,
- cudaStream_t stream
Restore auxiliary conv states into a single resident slot, zeroing any the capture lacks.
- Parameters:
saved – Tensors from captureAuxConvStates, or empty to start the slot from zero.
slot – Target resident state-pool row.
stream – CUDA stream.
-
void clearAuxConvStates(int32_t slot, cudaStream_t stream)#
Zero every auxiliary conv state for a single resident slot.
- Parameters:
slot – Resident state-pool row to clear.
stream – CUDA stream.
-
Config const &getConfig() const noexcept#
The configuration this manager was built with.
- Returns:
Reference to the stored configuration.
-
struct KVHeadDimGroupView#
Minimal read-only view of one pre-computed KV head-dim group.
Exposes just what callers (e.g. SpecDecode base verification) need to launch a batched per-layer kernel: a device-resident
KVLayerInfoarray plus the dispatch parameters. Internal bookkeeping (hostInfos,deviceScratchInfos, etc.) stays private.Public Members
-
kernel::KVLayerInfo const *deviceLayerInfos#
Device pointer to layer-info array (size == numLayers)
-
int32_t numLayers#
Number of KV layers in this group.
-
int32_t headDim#
Head dimension shared by all layers in this group.
-
int32_t maxKVHeads#
Maximum numKVHeads across layers in this group.
-
kernel::KVLayerInfo const *deviceLayerInfos#
-
HybridCacheManager() noexcept = default#
-
struct KVHeadDimGroupView
Minimal read-only view of one pre-computed KV head-dim group.
Exposes just what callers (e.g. SpecDecode base verification) need to launch a batched per-layer kernel: a device-resident
KVLayerInfoarray plus the dispatch parameters. Internal bookkeeping (hostInfos,deviceScratchInfos, etc.) stays private.Public Members
-
kernel::KVLayerInfo const *deviceLayerInfos
Device pointer to layer-info array (size == numLayers)
-
int32_t numLayers
Number of KV layers in this group.
-
int32_t headDim
Head dimension shared by all layers in this group.
-
int32_t maxKVHeads
Maximum numKVHeads across layers in this group.
-
kernel::KVLayerInfo const *deviceLayerInfos