DynamicBatchConfig#

class tensorrt_llm.llmapi.DynamicBatchConfig( *, enable_batch_size_tuning: bool = True, enable_max_num_tokens_tuning: bool = False, dynamic_batch_moving_average_window: int = 128, )[source]#

Bases: StrictBaseModel, PybindMirror

Dynamic batch configuration.

Controls how batch size and token limits are dynamically adjusted at runtime.

field dynamic_batch_moving_average_window: int = 128#: The window size for moving average of input and output length which is used to calculate dynamic batch size and max num tokens

field enable_batch_size_tuning: bool = True#: Controls if the batch size should be tuned dynamically

field enable_max_num_tokens_tuning: bool = False#: Controls if the max num tokens should be tuned dynamically

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

classmethod from_pybind( pybind_instance: PybindMirror, ) → T#

Construct an instance of the given class from the fields in the given pybind class instance.

Parameters:

cls – Type of the class to construct, must be a subclass of pydantic BaseModel
pybind_instance – Instance of the pybind class to construct from its fields

Notes

When a field value is None in the pybind class, but it’s not optional and has a default value in the BaseModel class, it would get the default value defined in the BaseModel class.

Returns:: Instance of the given class, populated with the fields of the given pybind instance

static get_pybind_enum_fields(pybind_class)#: Get all the enum fields from the pybind class.

static get_pybind_variable_fields(config_cls)#: Get all the variable fields from the pybind class.

static maybe_to_pybind(ins)#

static mirror_pybind_enum(pybind_class)#: Mirror the enum fields from the pybind class to the Python class.

static mirror_pybind_fields(pybind_class)#

Class decorator that ensures Python class fields mirror those of a C++ class.

Parameters:: pybind_class – The C++ class whose fields should be mirrored
Returns:: A decorator function that validates field mirroring

static pybind_equals(obj0, obj1)#: Check if two pybind objects are equal.