SchedulerConfig#

class tensorrt_llm.llmapi.SchedulerConfig( *, capacity_scheduler_policy: CapacitySchedulerPolicy = CapacitySchedulerPolicy.GUARANTEED_NO_EVICT, context_chunking_policy: ContextChunkingPolicy | None = None, dynamic_batch_config: DynamicBatchConfig | None = None, waiting_queue_policy: WaitingQueuePolicy = WaitingQueuePolicy.FCFS, use_python_scheduler: bool = False, enable_prefix_aware_scheduling: bool = True, )[source]#

Bases: StrictBaseModel, PybindMirror

field capacity_scheduler_policy: CapacitySchedulerPolicy = CapacitySchedulerPolicy.GUARANTEED_NO_EVICT#: The capacity scheduler policy to use

field context_chunking_policy: ContextChunkingPolicy | None = None#: The context chunking policy to use

field dynamic_batch_config: DynamicBatchConfig | None = None#: The dynamic batch configuration used by the scheduler to adjust batch size and token limits at runtime. This option currently has no effect since the scheduler does not support dynamic batch adjustments in the pytorch backend.

field enable_prefix_aware_scheduling: bool = True#: Use KV prefix-reuse estimates for scheduler admission, duplicate-request deferral, and token-budget decisions. This is orthogonal to kv_cache_config.enable_block_reuse.

field use_python_scheduler: bool = False#: Use pure-Python scheduler instead of C++ scheduler.

field waiting_queue_policy: WaitingQueuePolicy = WaitingQueuePolicy.FCFS#: The waiting queue scheduling policy

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

classmethod from_pybind( pybind_instance: PybindMirror, ) → T#

Construct an instance of the given class from the fields in the given pybind class instance.

Parameters:

cls – Type of the class to construct, must be a subclass of pydantic BaseModel
pybind_instance – Instance of the pybind class to construct from its fields

Notes

When a field value is None in the pybind class, but it’s not optional and has a default value in the BaseModel class, it would get the default value defined in the BaseModel class.

Returns:: Instance of the given class, populated with the fields of the given pybind instance

static get_pybind_enum_fields(pybind_class)#: Get all the enum fields from the pybind class.

static get_pybind_variable_fields(config_cls)#: Get all the variable fields from the pybind class.

static maybe_to_pybind(ins)#

static mirror_pybind_enum(pybind_class)#: Mirror the enum fields from the pybind class to the Python class.

static mirror_pybind_fields(pybind_class)#

Class decorator that ensures Python class fields mirror those of a C++ class.

Parameters:: pybind_class – The C++ class whose fields should be mirrored
Returns:: A decorator function that validates field mirroring

static pybind_equals(obj0, obj1)#: Check if two pybind objects are equal.