Encoder Embedding Cache#
-
class EncoderEmbeddingCache#
Content-addressed GPU cache for encoder (ViT / audio) output embeddings.
Saves encoded embeddings keyed by a 128-bit hash of the raw media bytes, allowing subsequent requests with identical media to skip the expensive encoder
infer()call. Eviction is LRU when the GPU memory budget is exceeded.Thread safety: inherits the single-writer contract from ContextCacheManager. No internal mutex.
Public Functions
-
explicit EncoderEmbeddingCache(int64_t maxBudgetBytes)#
Construct with a GPU memory budget in bytes. A budget of 0 disables the cache.
- std::optional<std::reference_wrapper<Tensor const>> lookup(
- Hash128 key
Look up a cached embedding by content hash. Returns a const reference to the cached tensor on hit, std::nullopt on miss. Updates last-access time on hit.
- void store( )#
Store an embedding under the given content hash by copying from the source tensor. Evicts LRU entries if the budget would be exceeded. The copy is enqueued on
stream.
- void storeSlice(
- Hash128 key,
- void const *devicePtr,
- int64_t numTokens,
- int64_t hiddenSize,
- nvinfer1::DataType dtype,
- cudaStream_t stream
Store a slice of an encoder output as a per-media-item cache entry.
- Parameters:
key – Content hash of the individual media item
devicePtr – Pointer into the encoder output buffer at the item’s byte offset
numTokens – Number of encoder output tokens for this item
hiddenSize – Hidden dimension (columns) of the embedding
dtype – Data type of the embedding elements
stream – CUDA stream for the device-to-device copy
-
void clear()#
Remove all entries and free GPU memory.
-
inline int64_t usedBytes() const noexcept#
Current GPU bytes used by cached embeddings.
-
inline size_t size() const noexcept#
Number of cached entries.
-
explicit EncoderEmbeddingCache(int64_t maxBudgetBytes)#
-
struct EncoderEmbeddingCacheEntry#
One cached encoder output embedding keyed by media content hash.