ExtendedRuntimePerfKnobConfig#

class tensorrt_llm.llmapi.ExtendedRuntimePerfKnobConfig( *, multi_block_mode: bool = True, enable_context_fmha_fp32_acc: bool = False, cuda_graph_mode: bool = False, cuda_graph_cache_size: int = 0, )[source]#

Bases: StrictBaseModel, PybindMirror

Configuration for extended runtime performance knobs.

field cuda_graph_cache_size: int = 0#: Number of cuda graphs to be cached in the runtime. The larger the cache, the better the perf, but more GPU memory is consumed.

field cuda_graph_mode: bool = False#: Whether to use CUDA graph mode.

field enable_context_fmha_fp32_acc: bool = False#: Whether to enable context FMHA FP32 accumulation.

field multi_block_mode: bool = True#: Whether to use multi-block mode.

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

classmethod from_pybind( pybind_instance: PybindMirror, ) → T#

Construct an instance of the given class from the fields in the given pybind class instance.

Parameters:

cls – Type of the class to construct, must be a subclass of pydantic BaseModel
pybind_instance – Instance of the pybind class to construct from its fields

Notes

When a field value is None in the pybind class, but it’s not optional and has a default value in the BaseModel class, it would get the default value defined in the BaseModel class.

Returns:: Instance of the given class, populated with the fields of the given pybind instance

static get_pybind_enum_fields(pybind_class)#: Get all the enum fields from the pybind class.

static get_pybind_variable_fields(config_cls)#: Get all the variable fields from the pybind class.

static maybe_to_pybind(ins)#

static mirror_pybind_enum(pybind_class)#: Mirror the enum fields from the pybind class to the Python class.

static mirror_pybind_fields(pybind_class)#

Class decorator that ensures Python class fields mirror those of a C++ class.

Parameters:: pybind_class – The C++ class whose fields should be mirrored
Returns:: A decorator function that validates field mirroring

static pybind_equals(obj0, obj1)#: Check if two pybind objects are equal.