SparseKvCacheOffloadConfig#
- class tensorrt_llm.llmapi.SparseKvCacheOffloadConfig(
- *,
- gpu_cache_factor: Annotated[float, Ge(ge=1.0)] = 2.0,
- host_history_size: Annotated[int, Ge(ge=0)] | None = None,
Bases:
StrictBaseModelMemory sizing for sparse attention KV cache offload.
With offload, the KV history a sparse attention layer selects from lives in host memory, and each request keeps a GPU cache of the units its selections name.
- field gpu_cache_factor: float = 2.0#
Size of each request’s GPU cache for offloaded attention history, in selections: the cache holds this many times the units one decode step selects, plus the unit being written, rounded up to whole pages. A larger cache keeps more history on the GPU between decode steps, so fewer units are fetched from host memory, at the cost of GPU memory per request.
- Constraints:
ge = 1.0
- field host_history_size: Annotated[int, Ge(ge=0)] | None = None#
Pinned host memory in bytes, per rank, for offloaded attention history and the staging buffers that publish it to the GPU. The KV cache manager holds the history pools within it, beside the GPU KV cache quota, which it does not lower; a size too small for the batches the manager must always hold, such as one max_seq_len request, is rejected. It is separate from kv_cache_config.host_cache_size, the host tier that evicted KV cache pages move to, and the two together must fit this rank’s share of available host memory and the pinned-memory limit. When unset, the history pools come first: they may take the host memory automatic sizing may pin, beside the staging buffers and, unless kv_cache_config.host_cache_size or a disk tier is set, room in the host tier for one max_seq_len request. Either way it is an upper bound: the history pools share their slot count with the GPU KV cache pools, so they pin only what the GPU quota’s slots pair with. When unset, the automatic host tier takes the host memory they leave, up to the GPU KV cache quota.
- __init__(**data: Any) None#
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.
self is explicitly positional-only to allow self as a field name.