Request Coordinator#
-
class RequestCoordinator#
The engine’s table of residents: the canonical owner of resident identity under the stepped control plane, and the single point where a StepResult becomes fact.
The runtime’s dense execution layout (which row a sequence occupies) is transient and runner-private; this table maps each live ResidentRef to its request record, its request, and how far its state is committed, and is updated only inside commit(). It lives as long as the session it describes – residents come and go, the table stays – and it is what the engine builds steps from: the context step of a newly admitted sequence covers its uncommitted prompt, a decode step advances every resident by one token from its committed length.
The ledger mirrors what the runtime executed, never what the plan asked: a step’s execution record (StepResult::execution) says how many positions each participant gained, and that is what commit() adds. The two differ under a visual-token pruner, whose context step adds fewer positions than the prompt range it was asked to cover; a ledger advanced from the plan would then build decode steps from a past length the runtime does not have.
Thread-free by design: the engine’s actor thread pumps it, and every transition is deterministic, so the whole scheduling state machine is testable without a thread.
Public Functions
- ResidentRef ref,
- std::shared_ptr<RequestRecord> record,
- std::unique_ptr<LLMGenerationRequest> request,
- AdmissionResult const &admitted
Seat one resident after its admission.
admittedsupplies the prompt length and the reused prefix the context step is built from; the request is kept for the compatibility comparison of later heads and for its stop strings.
-
ScheduledSequenceDescriptor contextStep(ResidentRef ref) const#
The context step of a resident that has not run one yet: its whole uncommitted prompt, sampling one token.
-
Progress progress(ResidentRef ref) const#
-
LLMGenerationRequest const &request(ResidentRef ref) const#
The request a resident was admitted with.
-
RequestRecord *findRecord(ResidentRef ref) noexcept#
The record of live resident
ref, or null when the table does not hold it.
- ScheduledSequenceDescriptor contextChunk(
- ResidentRef ref,
- int32_t queryLength
One prompt chunk of a resident whose context has not committed:
queryLengthpositions from its committed length, sampling exactly when the chunk ends the prompt (the final chunk). Requires 0 < queryLength <= promptLength - committed.
- void decodeStep(
- std::vector<ResidentRef> const &residents,
- std::vector<ScheduledSequenceDescriptor> &plan
Append to
planone decode step overresidents(the session’s live set, in its order): one query token each, past its committed length.
- std::vector<TerminalOutcome> commit(
- StepResult const &result,
- RuntimeSession const &session
Fold one StepResult into the table: advance every executed sequence’s committed length by what the runtime executed for it, mark the context committed when the executed context work sampled (its final chunk; an earlier chunk only advances committed), and turn every finished snapshot into a TerminalOutcome (materializing through
session). Refs are stable handles, so there is nothing to re-key; deltas need no handling either – token delivery flows through the runtime’s own stream channels. All-or-nothing: a throw (a ref that is not the live owner, an execution record that started from a past length the ledger does not hold, a materialize failure) leaves the table exactly as it was.
- std::vector<TerminalOutcome> retireEverything(
- FinishReason reason,
- std::string errorMessage = {}
Every remaining resident owes its caller an outcome; a dead session pays that debt here, or a cancelling shutdown does.
reasonis what each outcome reports, anderrorMessage, for a failure, why – the caller’s message reaches every retired resident.
-
std::vector<Released> releasePromptIncomplete()#
Take every resident whose prompt’s final chunk has not committed – one that has sampled no token – off the table, with its record and request, so the caller can admit it again into a fresh session.
- bool describes(
- std::vector<ResidentRef> const &residents
True when
residents– the session’s live set – is exactly the set this table holds. Checked after every step: a sequence the session holds that the table does not, or the reverse, means the two ledgers have diverged and nothing built from either can be trusted.
-
inline bool empty() const noexcept#
-
inline int32_t size() const noexcept#
-
LLMGenerationRequest const *anyRequest() const noexcept#
Any resident’s request, for the compatibility comparison of a head against the batch: every resident was admitted through the same gate, so they agree on the compared fields and any one stands for all. Null when nothing is resident.
-
int32_t mostUrgentPriority() const noexcept#
The lowest priority value among the residents’ requests (the most urgent one); INT32_MAX when nothing is resident.
-
struct Progress#
Where one resident stands, for a planner composing compact steps.
Public Members
-
SequenceIdentity identity#
-
int32_t promptLength = {0}#
The prompt as the runtime tokenized it.
-
int32_t committed = {0}#
Positions with state behind them (see Entry::committed).
-
bool contextCommitted = {false}#
The prompt’s final chunk ran: the resident decodes from here on.
-
SequenceIdentity identity#
-
struct Released#
A resident taken off the table without an outcome, to be queued again.
Public Members
-
ResidentRef ref#
-
std::shared_ptr<RequestRecord> record#
-
std::unique_ptr<LLMGenerationRequest> request#
-
ResidentRef ref#
-
struct TerminalOutcome#
What commit() decided a record’s fate is. The engine publishes these (terminate, counters, pending-count release) – publication is actor-side policy, the decision is made here.
Public Members
-
std::shared_ptr<RequestRecord> record#
-
FinishReason reason = {FinishReason::kNotFinished}#
-
std::optional<LLMGenerationResponse> response#
Present for a completed sequence: the materialized response.
-
std::string errorMessage#
For a failed sequence: why, when the retiring party knows (a failed session names its failure). Empty for a sequence that failed inside its own step.
-
std::shared_ptr<RequestRecord> record#
-
struct TerminalOutcome
What commit() decided a record’s fate is. The engine publishes these (terminate, counters, pending-count release) – publication is actor-side policy, the decision is made here.
Public Members
-
std::shared_ptr<RequestRecord> record
-
FinishReason reason = {FinishReason::kNotFinished}
-
std::optional<LLMGenerationResponse> response
Present for a completed sequence: the materialized response.
-
std::string errorMessage
For a failed sequence: why, when the retiring party knows (a failed session names its failure). Empty for a sequence that failed inside its own step.
-
std::shared_ptr<RequestRecord> record
-
struct Progress
Where one resident stands, for a planner composing compact steps.
Public Members
-
SequenceIdentity identity
-
int32_t promptLength = {0}
The prompt as the runtime tokenized it.
-
int32_t committed = {0}
Positions with state behind them (see Entry::committed).
-
bool contextCommitted = {false}
The prompt’s final chunk ran: the resident decodes from here on.
-
SequenceIdentity identity
-
struct Released
A resident taken off the table without an outcome, to be queued again.
Public Members
-
ResidentRef ref
-
std::shared_ptr<RequestRecord> record
-
std::unique_ptr<LLMGenerationRequest> request
-
ResidentRef ref