Step Planner

class StepPlanner

Chooses the work of one IFB step from the residents’ ledger state.

order(x) = (priority, arrival) under SchedulingPolicy::kPriority, else arrival.

mixed: decode rows take one token each first; with any decode row the step budget is maxNumTokens and the per-sequence chunk cap maxChunkTokens, otherwise T_max and S_in. prefillFirst: with any prompt-incomplete resident only prefill is scheduled (budget T_max, cap S_in), else every resident decodes; under kPriority that prefill step holds only the prompts of the best (lowest) pending priority value, and the others wait for a later prefill step. Prefills are then filled in order: n = min(remaining, cap, S_in, budget); a non-final chunk is trimmed back to a page boundary of committed + n; a resident left with n <= 0 is skipped so a later, shorter prompt may still fit. The first prompt in order is never skipped: admission bounds a prompt by S_in, and beside decode rows the budget and maxChunkTokens are at least P, so its chunk completes the prompt or reaches the next page boundary.

Pause (mixed, priority policy, preemption=pause): h is the prompt-incomplete resident first in order; residents with a larger priority value than h are paused and the plan is made over the rest, so when no active resident decodes h gets the whole-prompt budget. The pause set is recomputed from the ledger on every call and empties once h’s final chunk commits. With maxPauseSteps > 0, a resident whose consecutivePausedSteps has reached it is not paused in this plan. Under kFcfs every request counts as priority 0, so nothing is paused.

plan() allocates nothing: the buffers are sized for maxBatchSize at construction and the returned plan is overwritten by the next call.

Public Functions

StepPlanner(

ResolvedSchedulingPolicy const &policy,

StepPlannerLimits const &limits

)

policy comes from resolveSchedulingPolicy() for an engine with limits, which validates its budgets.

Throws:

std::runtime_error – when its mode is not mixed or prefillFirst.

StepPlan const &plan(ResidentState const *residents, int32_t count)

Throws:

std::invalid_argument – when count exceeds maxBatchSize or a prompt-incomplete resident has no remaining prompt.

inline StepPlan const &plan(

std::vector<ResidentState> const &residents

)

struct ResidentState

The ledger view of one resident that plan() reads.

Public Members

int32_t priority = {0}

Lower value = higher priority. Ignored under SchedulingPolicy::kFcfs.

uint64_t arrival = {0}

Monotonic admission-queue arrival stamp; unique per request.

int32_t promptLength = {0}

int32_t committed = {0}

Tokens already in KV and recurrent state (the frontier).

bool contextCommitted = {false}

The final prompt chunk has committed: the resident decodes. While false, the remaining prompt promptLength - committed must be positive.

int32_t consecutivePausedSteps = {0}

Steps in a row that withheld this resident under preemption=pause, up to the step being planned.

struct StepPlannerLimits

Engine build limits the planner budgets against.

Public Members

int32_t maxBatchSize = {0}

B: resident slots, and the bound on the residents passed to plan().

int32_t maxInputLength = {0}

S_in: per-sequence prefill tokens of one step. T_max = B x S_in.

int32_t kvPageSize = {0}

P: non-final chunks end at a multiple of P.

struct PlannedSequence

Public Members

int32_t resident = {0}

Index of the resident in the array passed to plan().

SequenceWork work = {SequenceWork::kDecode}

kContext for a prompt chunk, kDecode for one decode token.

int32_t pastLength = {0}

Equals the resident’s committed.

int32_t queryLength = {0}

bool selectLastTokenLogits = {false}

True for decode rows and for the chunk that completes the prompt; a non-final chunk only advances committed.

struct StepPlan

A planned step. With residents it always names at least one of them: the first unpaused resident in order gets a decode row or a chunk of at least one token (see StepPlanner). The session decides how a plan runs.

Public Members

std::vector<PlannedSequence> sequences

All prefill chunks in scheduling order, then all decode rows in scheduling order (the compact layout).

std::vector<int32_t> paused

Residents withheld by preemption=pause, ascending index. Residents left out for any other reason (prefill-first decodes, prefills that got no budget) are simply absent from sequences.

int32_t numPrefill = {0}

Np, Nd, T_p and T.

int32_t numDecode = {0}

int32_t prefillTokens = {0}

int32_t totalTokens = {0}