CacheTransceiverConfig

class tensorrt_llm.llmapi.CacheTransceiverConfig(

*,

backend: Literal['DEFAULT', 'UCX', 'NIXL', 'MOONCAKE', 'MPI'] | None = None,

transceiver_runtime: Literal['CPP', 'PYTHON', 'auto'] | None = 'PYTHON',

max_tokens_in_buffer: int | None = None,

kv_transfer_timeout_ms: Annotated[int, Gt(gt=0)] | None = 60000,

kv_transfer_sender_future_timeout_ms: Annotated[int, Gt(gt=0)] | None = 1000,

kv_transfer_poll_interval_ms: Annotated[int, Gt(gt=0)] | None = 5000,

kv_cache_bounce_size_mb: Annotated[int, Ge(ge=0)] = 0,

enable_pipelined_transfer: bool = False,

agent_bounce_buffer_enable: bool = False,

agent_bounce_params: Dict[str, str] | None = None,

)

Bases: StrictBaseModel, PybindMirror

Configuration for the cache transceiver.

field agent_bounce_buffer_enable: bool = False

DEPRECATED: no-op kept so existing configs still parse; it will be removed in a future release. kv_cache_bounce_size_mb > 0 alone enables the C++ transfer-agent bounce buffer, and 0 disables it.

field agent_bounce_params: Dict[str, str] | None = None

Expert tuning knobs for the C++ transfer-agent bounce pipeline; see tensorrt_llm/_torch/disaggregation/nixl/bounce_knobs.py for the valid keys. Byte-valued knobs accept a KB/MB/GB suffix (e.g. ‘32MB’). Precedence: this dict > environment variable > built-in default. Requires kv_cache_bounce_size_mb > 0. request_timeout_ms and the effective chunk size (max_chunk_size, capped by the buffer size) must match between context and generation. max_average_descriptor_size=0 stops this worker from sending through the buffer, but it still allocates the buffer and receives bounced writes; set kv_cache_bounce_size_mb=0 to turn the feature off.

field backend: Literal['DEFAULT', 'UCX', 'NIXL', 'MOONCAKE', 'MPI'] | None = None

The communication backend type to use for the cache transceiver.

field enable_pipelined_transfer: bool = False

Transfer each completed prefill chunk’s KV cache while later chunks compute. Requires Python NIXL, generation-first scheduling, chunked prefill, pipeline_parallel_size=1, context_parallel_size=1 on both peers, beam_width=1, no Mamba/hybrid cache, and block reuse disabled or set to all_reusable. Invalid static settings fail at startup; per-request constraints reject the request.

field kv_cache_bounce_size_mb: int = 0

Size in MiB of the KV-cache bounce buffer in the C++ NIXL transfer agent; 0 (default) turns it off and KV fragments are written directly with standard NIXL. A positive value allocates one buffer of this size per rank, shared by send and receive. Use a power of two such as 256, 512 or 1024; for other values only the largest power of two that fits is used. For an eligible write, the sender gathers the fragments into the buffer and sends them in chunks, and the receiver scatters them into its KV cache. By default a write is eligible only with >= 1024 descriptors averaging <= 16 KiB (typical when context and generation split attention heads differently); agent_bounce_params changes that. Set it on both context and generation servers; if either side has 0, the pair uses standard NIXL. If either side’s buffer is smaller than max_chunk_size (32 MiB by default), both sides must use the same size, or the pair uses standard NIXL. Requires the Python (v2) transceiver with the C++ NIXL agent; on a build without bounce support (libzmq) a positive value fails at startup. The C++ transceiver and the pure-Python NIXL agent ignore it with a warning.

Constraints:

• ge = 0

field kv_transfer_poll_interval_ms: Annotated[int, Gt(gt=0)] | None = 5000

Bounded wait interval in milliseconds for polling KV transfer progress when active transfers block disaggregated admission.

field kv_transfer_sender_future_timeout_ms: Annotated[int, Gt(gt=0)] | None = 1000

Duration in milliseconds of each bounded sender future wait slice while polling KV transfer completion. It does not set the overall transfer deadline.

field kv_transfer_timeout_ms: Annotated[int, Gt(gt=0)] | None = 60000

KV cache transfer timeout in milliseconds. Blocking sender waits use it as an absolute deadline; blocking receive task waits use it per task. The Python V2 transceiver requires a finite value; None remains available to other runtimes. It is distinct from the sender future wait slice.

field max_tokens_in_buffer: int | None = None

The max number of tokens the transfer buffer can fit.

field transceiver_runtime: Literal['CPP', 'PYTHON', 'auto'] | None = 'PYTHON'

The runtime implementation. ‘PYTHON’ (default) selects the Python transceiver, while ‘CPP’ selects the C++ transceiver and requires kv_cache_config.use_kv_cache_manager_v2=False with a model that supports V1. ‘auto’ adopts the model’s preferred runtime when it declares one; otherwise it selects the Python transceiver, falling back to the C++ transceiver only when this config itself rules it out (non-NIXL backend or a null kv_transfer_timeout_ms) — any other incompatibility fails at transceiver creation. The fallback is decided independently on each server and is only logged, not surfaced, so keep context and generation server configurations consistent. None is equivalent to ‘CPP’. ‘auto’ is resolved on the PyTorch backend’s standard model-loading path.

__init__(**data: Any) → None

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

validator coerce_agent_bounce_params_to_str  »  agent_bounce_params

Coerce agent_bounce_params values to strings (YAML often yields ints/bools).

classmethod from_pybind(

pybind_instance: PybindMirror,

) → T

Construct an instance of the given class from the fields in the given pybind class instance.

Parameters:

• cls – Type of the class to construct, must be a subclass of pydantic BaseModel

• pybind_instance – Instance of the pybind class to construct from its fields

Notes

When a field value is None in the pybind class, but it’s not optional and has a default value in the BaseModel class, it would get the default value defined in the BaseModel class.

Returns:

Instance of the given class, populated with the fields of the given pybind instance

static get_pybind_enum_fields(pybind_class)

Get all the enum fields from the pybind class.

static get_pybind_variable_fields(config_cls)

Get all the variable fields from the pybind class.

static maybe_to_pybind(ins)

static mirror_pybind_enum(pybind_class)

Mirror the enum fields from the pybind class to the Python class.

static mirror_pybind_fields(pybind_class)

Class decorator that ensures Python class fields mirror those of a C++ class.

Parameters:

pybind_class – The C++ class whose fields should be mirrored

Returns:

A decorator function that validates field mirroring

static pybind_equals(obj0, obj1)

Check if two pybind objects are equal.

validator validate_bounce_config  »  all fields