PromptTokenCacheConfig#

class tensorrt_llm.llmapi.PromptTokenCacheConfig(
*,
path: str,
max_bytes: Annotated[int, Gt(gt=0)],
)[source]#

Bases: StrictBaseModel

Token ids of earlier prompts, shared by all serving frontends of a node.

A multi-turn prompt repeats the previous turn’s prompt; frontends that support the cache tokenize only the text after the longest cached prefix. The ids always equal a full tokenization.

field max_bytes: Annotated[int, Gt(gt=0)] [Required]#

Cap on the space the entries take in path, each counted in whole file system blocks (pages on a tmpfs); the least recently used entries are evicted first.

Constraints:
  • gt = 0

field path: str [Required]#

Directory for the cache on a node-local file system such as a tmpfs, created if missing. Every frontend of the server must use the same directory.

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.