Scheduling Policy

struct SchedulingOptions

The in-flight batching scheduling options, named as the bindings and the server name them. B, S_in and P are the engine’s build batch size, its input length and the KV page size; T_max = B x S_in. See resolveSchedulingPolicy() for the constraints.

Public Members

IfbSchedulingMode ifbSchedulingMode = {IfbSchedulingMode::kAuto}

int32_t maxNumTokens = {0}

Token budget of a mixed step with decode rows, in [B + P, T_max]. 0 selects min(roundUp_P(512 + B), roundDown_P(T_max)).

int32_t maxChunkTokens = {0}

Per-sequence prefill budget of a mixed step with decode rows: a multiple of P in [P, maxNumTokens - B]. 0 selects min(512, roundDown_P(maxNumTokens - B)). Either is then capped at max(P, roundDown_P(S_in)), the most one prompt chunk holds.

SchedulingPolicy schedulingPolicy = {SchedulingPolicy::kFcfs}

PreemptionMode preemption = {PreemptionMode::kNone}

int32_t priorityReservedSlots = {0}

Slots held back for requests with priority < 0: any other request is admitted only while more than this many slots are free.

int32_t maxPauseSteps = {0}

Under PreemptionMode::kPause, the most steps in a row a resident is paused: after this many it is not paused in the next step even while a more urgent prompt is unfinished. That guarantees a decode row only to a decoding resident; one with an unfinished prompt competes for the prompt budget in priority order. 0 = no limit.

struct StepBudgets

The budgets of a mixed step with decode rows, as the planner applies them.

Public Members

int32_t maxNumTokens = {0}

int32_t maxChunkTokens = {0}

struct ResolvedSchedulingPolicy

The policy the engine runs with; see RequestEngine::resolvedPolicy() for when it is resolved.

Public Members

IfbSchedulingMode mode = {IfbSchedulingMode::kPrefillFirst}

kMixed, kPrefillFirst or kSingleCohort; never kAuto.

ModeSource modeSource = {ModeSource::kRequested}

std::string reason

Why auto chose the mode; empty for a requested mode.

SchedulingPolicy schedulingPolicy = {SchedulingPolicy::kFcfs}

PreemptionMode preemption = {PreemptionMode::kNone}

As applied: kNone when auto resolved to prefillFirst.

StepBudgets budgets

Zero outside mixed, whose prefill steps are bounded by the engine (T_max tokens, S_in per sequence) instead.

int32_t priorityReservedSlots = {0}

int32_t maxPauseSteps = {0}

0 = no limit, and 0 whenever preemption is kNone.

bool compactExecution = {false}

Whether the session executes compact steps.

std::vector<std::string> warnings

Options the engine accepted but does not apply as given, one sentence each; logged at startup.

void trt_edgellm::rt::scheduler::validateSchedulingOptions(

SchedulingOptions const &options,

int32_t maxResidents

)

Refuse what could never apply, or would leave requests waiting forever, whatever the deployment: priorityReservedSlots outside [0, maxResidents - 1]; reserved slots or pause without the priority policy; a negative maxPauseSteps, or a positive one without pause; pause with an explicit prefillFirst or singleCohort. These checks need no engine facts, so a caller can make them before the engine is built.

Parameters:

maxResidents – EngineConfig::maxBatchSize, which bounds priorityReservedSlots.

Throws:

std::invalid_argument – naming the option and the alternative.

ResolvedSchedulingPolicy trt_edgellm::rt::scheduler::resolveSchedulingPolicy(

SchedulingOptions const &options,

int32_t maxResidents,

SessionCapabilities const &capabilities

)

Resolve options against what the deployment can execute.

B, S_in and P come from capabilities; maxResidents (EngineConfig::maxBatchSize) bounds priorityReservedSlots.

Throws:

std::invalid_argument – naming the option and the alternative when an option could never apply or the deployment cannot serve it: what validateSchedulingOptions() refuses; a budget out of range (in every mode); mixed without compact execution or without room for B + P tokens. Under singleCohort, compactExecution is false whatever capabilities reports. ResolvedSchedulingPolicy::warnings reports what is accepted but not applied as given.

char const *trt_edgellm::rt::scheduler::executionName(

ResolvedSchedulingPolicy const &policy

) noexcept

“compact” when policy executes compact steps, else “single-cohort”.

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::rt::scheduler::toString” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:

- char const *toString(IfbSchedulingMode mode) noexcept
- char const *toString(ModeSource source) noexcept
- char const *toString(PreemptionMode mode) noexcept
- char const *toString(SchedulingPolicy policy) noexcept

template<typename Enum>
Enum trt_edgellm::rt::scheduler::parseEnum(

std::string const &name

)

Parse a name toString() produces for IfbSchedulingMode, SchedulingPolicy or PreemptionMode.

Throws:

std::invalid_argument – naming the option and the accepted names.