System Prompt KV Cache

struct SystemPromptKVCache

Structure to hold cached system prompt and its KV cache.

Public Members

std::string systemPrompt

The system prompt text.

std::vector<tokenizer::Rank> tokenizedPrompt

Tokenized version of the system prompt.

std::vector<rt::Tensor> kvCacheLayers

Per-layer KV cache tensors for the system prompt.

std::vector<rt::Tensor> recurrentStateContents

Cached recurrent states for hybrid layers (empty if not applicable)

std::vector<rt::Tensor> convStateContents

Cached conv states for hybrid layers (empty if not applicable)

struct SystemPromptReuseResult

Result of reconciling a request’s input tokens against a previously-captured system-prompt KV cache.

Public Members

int32_t reuseKVCacheLength

Number of cached tokens to reuse (recorded for KV cache reuse)

std::vector<int32_t> tokenIds

Remaining input tokens still to be prefilled.

int32_t effectivePrefillLength

Number of tokens left to prefill (== tokenIds.size())