KVEventsConfig#

class tensorrt_llm.llmapi.KVEventsConfig(
*,
enable_kv_cache_events: bool = False,
publisher: Literal['null', 'zmq'] | None = None,
endpoint: Annotated[str, MinLen(min_length=1)] = 'tcp://*:5557',
replay_endpoint: Annotated[str | None, MinLen(min_length=1)] = None,
buffer_steps: Annotated[int, Gt(gt=0)] = 10000,
hwm: Annotated[int, Gt(gt=0)] = 100000,
max_queue_size: Annotated[int, Gt(gt=0)] = 100000,
topic: str = '',
)[source]#

Bases: StrictBaseModel

Configuration for streaming (push-based) KV cache event publishing.

field buffer_steps: int = 10000#

Number of previously published batches retained for replay.

Constraints:

  • gt = 0

field enable_kv_cache_events: bool = False#

Whether to produce and publish KV cache events over the streaming (push) path.

field endpoint: str = 'tcp://*:5557'#

Base ZeroMQ endpoint the publisher binds. Each attention-DP rank binds base_port+rank, so co-located engines (e.g. disaggregated prefill and decode on one host) must use distinct base ports.

Constraints:

  • min_length = 1

field hwm: int = 100000#

ZeroMQ publisher socket high-water mark. 0 means unlimited in ZeroMQ, so it is disallowed here.

Constraints:

  • gt = 0

field max_queue_size: int = 100000#

Maximum number of batches queued for background publishing. Must be positive; 0 would make the queue unbounded.

Constraints:

  • gt = 0

field publisher: Literal['null', 'zmq'] | None = None#

Publisher implementation. Defaults to ‘zmq’ when events are enabled and ‘null’ otherwise.

field replay_endpoint: str | None = None#

Optional base ZeroMQ endpoint used to replay KV cache events. Ranks apply the same global base_port+rank convention as endpoint. Only ranks sharing a host contend for a port, so the two base ports must be at least ranks-per-host apart or a rank’s replay bind collides with another rank’s publish bind on that host.

Constraints:

  • min_length = 1

field topic: str = ''#

ZeroMQ subscription topic used for KV cache event batches.

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.