Hybrid Cache Manager#

class HybridCacheManager#

Top-level cache manager for hybrid Attention + Mamba architectures.

Routes cache access by absolute decoder-layer index to the appropriate sub-manager (KVCacheManager for attention layers, MambaCacheManager for recurrent layers). Owns the shared device KV cache lengths tensor and provides unified compaction, prompt cache capture / restore, and batch management APIs that span both sub-managers.

Public Types

enum class LayerType#

Type of a single decoder layer.

Values:

enumerator kAttention#
enumerator kMamba#

Public Functions

HybridCacheManager() noexcept = default#

Default constructor (no allocation)

HybridCacheManager(Config const &config, cudaStream_t stream)#

Construct and initialise sub-managers and routing tables.

Builds absolute-to-local index mapping for each sub-manager, constructs KVCacheManager and MambaCacheManager, and allocates the shared device KV-cache-lengths tensor (zero-initialised).

Parameters:
  • config – Cache configuration with per-layer type routing

  • stream – CUDA stream for allocation and memset

~HybridCacheManager() noexcept#

Destructor.

HybridCacheManager(HybridCacheManager const&) = delete#

Deleted copy constructor.

HybridCacheManager &operator=(HybridCacheManager const&) = delete#

Deleted copy assignment.

Returns:

Reference to this

HybridCacheManager(HybridCacheManager&&) noexcept#

Move constructor.

HybridCacheManager &operator=(HybridCacheManager&&) noexcept#

Move assignment operator.

Returns:

Reference to this

inline int32_t numLayers() const noexcept#

Total number of decoder layers (attention + Mamba).

Returns:

Number of layers (== size of the configured layer-type list).

LayerType getLayerType(int32_t absLayerIdx) const#

Type (attention or Mamba/recurrent) of a given absolute layer index.

Parameters:

absLayerIdx – Absolute decoder-layer index in [0, numLayers()).

Returns:

The layer’s type.

rt::Tensor &getCombinedKVCache(int32_t absLayerIdx)#

Get the combined KV-cache page pool for a given absolute layer index (must be an attention layer).

Parameters:

absLayerIdx – Absolute decoder-layer index.

Returns:

Reference to the page pool [2, numPages, kTOKENS_PER_PAGE, numKVHeads, headDim].

rt::Tensor const &getCombinedKVCache(int32_t absLayerIdx) const#
std::pair<rt::Tensor, rt::Tensor> getSeparateKVCache(
int32_t absLayerIdx
)#

Get the K-half and V-half of a given attention layer’s pool as separate tensor views.

Parameters:

absLayerIdx – Absolute decoder-layer index.

Returns:

{kView, vView}, each shaped [maxBatch, capPadded, numKVHeads, headDim].

KVLayerStorageMetadata getKVLayerStorageMetadata(
int32_t absLayerIdx
) const#

Describe the physical pool and logical sequence geometry used by an attention layer.

rt::Tensor &getRecurrentState(int32_t absLayerIdx)#

Get the recurrent state for a given absolute layer index (must be a Mamba layer).

Parameters:

absLayerIdx – Absolute decoder-layer index.

Returns:

Reference to the per-layer recurrent state tensor.

rt::Tensor &getConvState(int32_t absLayerIdx, int32_t stateIdx = 0)#

Get a conv state for a given absolute layer index.

Parameters:
  • absLayerIdx – Absolute decoder-layer index.

  • stateIdx – 0 for the layer’s own mixer state, which requires a Mamba layer; above that, one of the configured auxiliary states on that layer.

Returns:

Reference to the per-slot conv state tensor.

int32_t numAuxConvStates() const noexcept#

Number of configured auxiliary conv states, across all layers.

Returns:

Count, zero for a model that declares none.

AuxConvStateConfig const &auxConvStateConfig(int32_t auxIdx) const#

The configuration one auxiliary conv state was allocated from.

Parameters:

auxIdx – Index in [0, numAuxConvStates()).

Returns:

The entry, whose absLayerIdx names the layer it belongs to.

rt::Tensor &getAuxConvState(int32_t auxIdx)#

Storage for one auxiliary conv state, shaped [maxBatchSize, convDim, convWidth].

Parameters:

auxIdx – Index in [0, numAuxConvStates()).

Returns:

Reference to the owned tensor.

bool hasIntermediateAuxConvStates() const noexcept#

Whether the auxiliary conv states have spec-verify snapshots.

Returns:

True when allocated, which follows the Mamba intermediate states.

rt::Tensor &getIntermediateAuxConvState(int32_t auxIdx)#

Per-token spec-verify snapshot of one auxiliary conv state.

Parameters:

auxIdx – Index in [0, numAuxConvStates()).

Returns:

Reference to the owned [maxBatchSize, maxIntermediateSeqLen, convDim, convWidth] tensor.

void reshapeIntermediateAuxConvStates(
int32_t activeBatchSize,
int32_t seqLen
)#

Reshape the auxiliary snapshots to a step’s extents.

Parameters:
  • activeBatchSize – Sequences in the step.

  • seqLen – Verify tokens per sequence.

void scatterAcceptedAuxConvStates(
rt::Tensor const &acceptLengths,
rt::Tensor const &stateIndices,
cudaStream_t stream,
rt::Tensor const *acceptedStateNodeIds = nullptr
)#

Copy each sequence’s last accepted snapshot into its resident slot.

Parameters:
  • acceptLengths – [activeBatchSize] INT32 accepted token count per sequence.

  • stateIndices – [activeBatchSize] INT32 resident slot per sequence.

  • stream – CUDA stream.

  • acceptedStateNodeIds – Tree verify: [activeBatchSize, maxAcceptLen] INT32 accepted node ids in path order. Null for chain verify, where the snapshot is acceptLengths[b] - 1.

KVCacheManager &getKVCacheManager() noexcept#

Direct access to the KV cache sub-manager.

Returns:

Reference to the KVCacheManager.

KVCacheManager const &getKVCacheManager() const noexcept#
MambaCacheManager &getMambaCacheManager() noexcept#

Direct access to the Mamba state sub-manager.

Returns:

Reference to the MambaCacheManager.

std::vector<KVHeadDimGroupView> getKVHeadDimGroups() const#

Read-only views of the pre-computed KV head-dim groups.

Uniform models return a single group; hybrid Gemma4-style models return one group per distinct head dim. The underlying KVLayerInfo arrays are owned by this manager and remain valid for its lifetime.

Returns:

Vector of group views (one per distinct head dim).

rt::Tensor &getKVCacheLengths() noexcept#

Get the shared device KV cache lengths tensor.

Returns:

Reference to the device tensor of shape [activeBatchSize].

void resetForNewSequences(
rt::Tensor const &reuseKVCacheLengths,
cudaStream_t stream
)#

Reset state for new sequences. Validates batch size, copies reuse lengths from host to device, and updates the “all empty” flag.

Parameters:
  • reuseKVCacheLengths – Host INT32 tensor with reuse lengths, shape [batchSize].

  • stream – CUDA stream.

void commitSequenceLength(
rt::Tensor const &newContextLengths,
cudaStream_t stream
)#

Commit sequence lengths after prefill (element-wise increment from a GPU tensor).

Parameters:
  • newContextLengths – GPU INT32 tensor of context lengths, shape [activeBatchSize].

  • stream – CUDA stream.

void commitSequenceLength(int32_t increment, cudaStream_t stream)#

Commit sequence lengths after decode (scalar increment for all active sequences).

Parameters:
  • increment – Scalar increment (typically 1).

  • stream – CUDA stream.

int32_t getActiveBatchSize() const noexcept#

Get the number of active sequences.

Returns:

Active batch size.

void setActiveBatchSize(int32_t newActiveBatchSize)#

Set active batch size (used after batch eviction).

Parameters:

newActiveBatchSize – New active batch size.

Throws:

std::runtime_error – if out of range [0, maxBatchSize].

void restoreActiveBatchSize(int32_t activeBatchSize) noexcept#

Restore a previously validated batch-size view from an exception guard.

void swapExecutionRows(
int32_t rowA,
int32_t rowB,
cudaStream_t stream
)#

Exchange the KV lengths of two execution rows. Self-inverse; the device counterpart of DecodingInferenceContext::swapExecutionRows, which a batch-one projection pairs it with so the projected row runs as row zero while every other row’s length is kept where it belongs.

Throws:

std::runtime_error – if either row is outside the active batch or the swap cannot be enqueued.

void appendSequence(int32_t committedLength, cudaStream_t stream)#

Append one sequence to the active batch, as the last execution row, with the KV length it already holds (committedLength: a reused prefix, or zero). The runtime session reserves the row with the sequence’s other reservations so the batch here follows the context’s from the seat on, prefilled or not. Strong exception guarantee: a throw leaves the batch as it was.

Throws:

std::runtime_error – when the batch is full or the device update cannot be enqueued.

void materializeExecutionLengths(
rt::Tensor const &executionLengths,
cudaStream_t stream
)#

Materialize the logical lengths for the next contiguous execution batch.

The source is indexed by execution row. This updates only the transient length input read by the engines; KV pages and recurrent state remain indexed by their resident slots.

bool getKVCacheAllEmpty() const noexcept#

Check if KV cache for all sequences is empty.

Returns:

True if no prefill has been committed yet.

void compactKVCacheLengths(
rt::Tensor const &batchMapping,
int32_t oldBatch,
int32_t newBatch,
cudaStream_t stream
)#

Compact execution-row-aligned KV lengths without moving resident slot state.

void clearResidentSlot(int32_t slot, cudaStream_t stream)#

Clear one retired resident row’s recurrent/convolution state without moving survivors.

std::vector<rt::Tensor> captureKVCache(
int32_t batchIdx,
int32_t sequenceLength,
cudaStream_t stream
)#

Capture KV cache for a single batch slot across all attention layers.

Parameters:
  • batchIdx – Batch slot to capture.

  • sequenceLength – Number of tokens to capture from the cache.

  • stream – CUDA stream.

Returns:

Vector of captured tensors (one per attention layer).

void restoreKVCache(
std::vector<rt::Tensor> const &saved,
int32_t batchIdx,
cudaStream_t stream
)#

Restore KV cache for a single batch slot across all attention layers.

Parameters:
  • saved – Previously captured KV cache tensors.

  • batchIdx – Target batch slot.

  • stream – CUDA stream.

std::vector<rt::Tensor> captureRecurrentStates(
int32_t batchIdx,
cudaStream_t stream
)#

Capture recurrent states for a single batch slot (delegates to MambaCacheManager).

Parameters:
  • batchIdx – Batch slot to capture.

  • stream – CUDA stream.

Returns:

Vector of captured tensors (one per recurrent layer).

std::vector<rt::Tensor> captureConvStates(
int32_t batchIdx,
cudaStream_t stream
)#

Capture conv states for a single batch slot (delegates to MambaCacheManager).

Parameters:
  • batchIdx – Batch slot to capture.

  • stream – CUDA stream.

Returns:

Vector of captured tensors (one per recurrent layer).

std::vector<rt::Tensor> captureAuxConvStates(
int32_t slot,
cudaStream_t stream
)#

Capture every auxiliary conv state for a single resident slot.

Parameters:
  • slot – Resident state-pool row to capture, as in state_indices.

  • stream – CUDA stream.

Returns:

One tensor per auxiliary state, in configured order; empty when there are none.

void restoreAuxConvStates(
std::vector<rt::Tensor> const &saved,
int32_t slot,
cudaStream_t stream
)#

Restore auxiliary conv states into a single resident slot, zeroing any the capture lacks.

Parameters:
  • saved – Tensors from captureAuxConvStates, or empty to start the slot from zero.

  • slot – Target resident state-pool row.

  • stream – CUDA stream.

void clearAuxConvStates(int32_t slot, cudaStream_t stream)#

Zero every auxiliary conv state for a single resident slot.

Parameters:
  • slot – Resident state-pool row to clear.

  • stream – CUDA stream.

Config const &getConfig() const noexcept#

The configuration this manager was built with.

Returns:

Reference to the stored configuration.

struct KVHeadDimGroupView#

Minimal read-only view of one pre-computed KV head-dim group.

Exposes just what callers (e.g. SpecDecode base verification) need to launch a batched per-layer kernel: a device-resident KVLayerInfo array plus the dispatch parameters. Internal bookkeeping (hostInfos, deviceScratchInfos, etc.) stays private.

Public Members

kernel::KVLayerInfo const *deviceLayerInfos#

Device pointer to layer-info array (size == numLayers)

int32_t numLayers#

Number of KV layers in this group.

int32_t headDim#

Head dimension shared by all layers in this group.

int32_t maxKVHeads#

Maximum numKVHeads across layers in this group.

struct KVHeadDimGroupView

Minimal read-only view of one pre-computed KV head-dim group.

Exposes just what callers (e.g. SpecDecode base verification) need to launch a batched per-layer kernel: a device-resident KVLayerInfo array plus the dispatch parameters. Internal bookkeeping (hostInfos, deviceScratchInfos, etc.) stays private.

Public Members

kernel::KVLayerInfo const *deviceLayerInfos

Device pointer to layer-info array (size == numLayers)

int32_t numLayers

Number of KV layers in this group.

int32_t headDim

Head dimension shared by all layers in this group.

int32_t maxKVHeads

Maximum numKVHeads across layers in this group.