PromptTokenCacheConfig#
- class tensorrt_llm.llmapi.PromptTokenCacheConfig(
- *,
- path: str,
- max_bytes: Annotated[int, Gt(gt=0)],
Bases:
StrictBaseModelToken ids of earlier prompts, shared by all serving frontends of a node.
A multi-turn prompt repeats the previous turn’s prompt; frontends that support the cache tokenize only the text after the longest cached prefix. The ids always equal a full tokenization.
- field max_bytes: Annotated[int, Gt(gt=0)] [Required]#
Cap on the space the entries take in
path, each counted in whole file system blocks (pages on a tmpfs); the least recently used entries are evicted first.- Constraints:
gt = 0
- field path: str [Required]#
Directory for the cache on a node-local file system such as a tmpfs, created if missing. Every frontend of the server must use the same directory.
- __init__(**data: Any) None#
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.
self is explicitly positional-only to allow self as a field name.