SchedulerConfig#
- class tensorrt_llm.llmapi.SchedulerConfig(
- *,
- capacity_scheduler_policy: CapacitySchedulerPolicy = CapacitySchedulerPolicy.GUARANTEED_NO_EVICT,
- context_chunking_policy: ContextChunkingPolicy | None = None,
- dynamic_batch_config: DynamicBatchConfig | None = None,
- waiting_queue_policy: WaitingQueuePolicy = WaitingQueuePolicy.FCFS,
- use_python_scheduler: bool = False,
- enable_prefix_aware_scheduling: bool = True,
- context_scheduling_policy: ContextSchedulingPolicy = ContextSchedulingPolicy.FCFS,
- context_starvation_iterations: Annotated[int, Gt(gt=0)] | None = None,
Bases:
StrictBaseModel,PybindMirror- field capacity_scheduler_policy: CapacitySchedulerPolicy = CapacitySchedulerPolicy.GUARANTEED_NO_EVICT#
The capacity scheduler policy to use
- field context_chunking_policy: ContextChunkingPolicy | None = None#
The context chunking policy to use
- field context_scheduling_policy: ContextSchedulingPolicy = ContextSchedulingPolicy.FCFS#
Order in which pending context requests are admitted: ‘fcfs’ (arrival order) or ‘shortest_uncached_first’ (fewest prompt tokens not covered by reusable KV cache first, which lowers TTFT under load at the cost of fairness). Only the KV cache manager V2 scheduler supports ‘shortest_uncached_first’.
- field context_starvation_iterations: Annotated[int, Gt(gt=0)] | None = None#
With context_scheduling_policy=’shortest_uncached_first’, chunked prefill and a max_num_tokens budget: a context request pending for more than this many scheduler iterations takes up to a quarter of max_num_tokens of each iteration’s budget ahead of the shortest-first order, oldest first. None disables it.
- field dynamic_batch_config: DynamicBatchConfig | None = None#
The dynamic batch configuration used by the scheduler to adjust batch size and token limits at runtime. This option currently has no effect since the scheduler does not support dynamic batch adjustments in the pytorch backend.
- field enable_prefix_aware_scheduling: bool = True#
Use KV prefix-reuse estimates for scheduler admission, duplicate-request deferral, and token-budget decisions. This is orthogonal to kv_cache_config.enable_block_reuse.
- field use_python_scheduler: bool = False#
Use pure-Python scheduler instead of C++ scheduler.
- field waiting_queue_policy: WaitingQueuePolicy = WaitingQueuePolicy.FCFS#
The waiting queue scheduling policy
- __init__(**data: Any) None#
Create a new model by parsing and validating input data from keyword arguments.
Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.
self is explicitly positional-only to allow self as a field name.
- classmethod from_pybind(
- pybind_instance: PybindMirror,
Construct an instance of the given class from the fields in the given pybind class instance.
- Parameters:
cls – Type of the class to construct, must be a subclass of pydantic BaseModel
pybind_instance – Instance of the pybind class to construct from its fields
Notes
When a field value is None in the pybind class, but it’s not optional and has a default value in the BaseModel class, it would get the default value defined in the BaseModel class.
- Returns:
Instance of the given class, populated with the fields of the given pybind instance
- static get_pybind_enum_fields(pybind_class)#
Get all the enum fields from the pybind class.
- static get_pybind_variable_fields(config_cls)#
Get all the variable fields from the pybind class.
- static maybe_to_pybind(ins)#
- static mirror_pybind_enum(pybind_class)#
Mirror the enum fields from the pybind class to the Python class.
- static mirror_pybind_fields(pybind_class)#
Class decorator that ensures Python class fields mirror those of a C++ class.
- Parameters:
pybind_class – The C++ class whose fields should be mirrored
- Returns:
A decorator function that validates field mirroring
- static pybind_equals(obj0, obj1)#
Check if two pybind objects are equal.