SchedulerConfig#

class tensorrt_llm.llmapi.SchedulerConfig( *, capacity_scheduler_policy: CapacitySchedulerPolicy = CapacitySchedulerPolicy.GUARANTEED_NO_EVICT, context_chunking_policy: ContextChunkingPolicy | None = None, dynamic_batch_config: DynamicBatchConfig | None = None, waiting_queue_policy: WaitingQueuePolicy = WaitingQueuePolicy.FCFS, use_python_scheduler: bool = False, enable_prefix_aware_scheduling: bool = True, )[source]#

Bases: StrictBaseModel, PybindMirror

field capacity_scheduler_policy: CapacitySchedulerPolicy = CapacitySchedulerPolicy.GUARANTEED_NO_EVICT#: The capacity scheduler policy to use

field context_chunking_policy: ContextChunkingPolicy | None = None#: The context chunking policy to use

field dynamic_batch_config: DynamicBatchConfig | None = None#: The dynamic batch config to use. This only applies for the TensorRT backend and cannot currently be used with the PyTorch backend.

field enable_prefix_aware_scheduling: bool = True#: Use KV prefix-reuse estimates for scheduler admission, duplicate-request deferral, and token-budget decisions. This is orthogonal to kv_cache_config.enable_block_reuse.

field use_python_scheduler: bool = False#: Use pure-Python scheduler instead of C++ scheduler.

field waiting_queue_policy: WaitingQueuePolicy = WaitingQueuePolicy.FCFS#: The waiting queue scheduling policy

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

classmethod from_pybind( pybind_instance: PybindMirror, ) → T#

Construct an instance of the given class from the fields in the given pybind class instance.

Parameters:

cls – Type of the class to construct, must be a subclass of pydantic BaseModel
pybind_instance – Instance of the pybind class to construct from its fields

Notes

When a field value is None in the pybind class, but it’s not optional and has a default value in the BaseModel class, it would get the default value defined in the BaseModel class.

Returns:: Instance of the given class, populated with the fields of the given pybind instance

static get_pybind_enum_fields(pybind_class)#: Get all the enum fields from the pybind class.

static get_pybind_variable_fields(config_cls)#: Get all the variable fields from the pybind class.

static maybe_to_pybind(ins)#

static mirror_pybind_enum(pybind_class)#: Mirror the enum fields from the pybind class to the Python class.

static mirror_pybind_fields(pybind_class)#

Class decorator that ensures Python class fields mirror those of a C++ class.

Parameters:: pybind_class – The C++ class whose fields should be mirrored
Returns:: A decorator function that validates field mirroring

static pybind_equals(obj0, obj1)#: Check if two pybind objects are equal.