Multi Stage Runtime#

class MultiStageRuntime#

The only C++ entry point for running a topology. It owns admission, each seat’s frozen Route, the lane streams and the step loop. It does not know that omni exists: every model family is a set of values in the config it was handed.

Drives one static cohort at a time: submit() up to the cohort limit, run(), then take().

Public Functions

MultiStageRuntime()#

~MultiStageRuntime()#

MultiStageRuntime(MultiStageRuntime const&) = delete#

MultiStageRuntime &operator=(MultiStageRuntime const&) = delete#

void load( std::string const &configRoot, SharedResources &shared, StageRegistry const *runners = nullptr )#

Parse, validate, build every stage and claim all device memory.

Throws naming the stage on any failure; a partially constructed runtime is never handed back. runners defaults to builtinRunners(); it is not read after load() returns.

void load( MultiStageConfig config, std::string const &configRoot, SharedResources &shared, StageRegistry const *runners = nullptr )#: load() for a config the caller parsed and may have adjusted. Paths still resolve against configRoot.

inline MultiStageConfig const &config() const noexcept#

IStage const *stage(std::string const &stageId) const noexcept#: The stage built for stageId, for a caller that knows its runner and wants more than IStage offers, such as a backbone’s aggregate metrics. Null for an id the config lacks.

RequestSlot *submit( PortBindings const &payload, Sinks const &sinks, std::string &error )#

Seat one request in the cohort: resolve the route, admit each stage.

Refused when the cohort is full or this request resolves a different route than the seated ones.

Pure CPU. Returns nullptr and fills error when the request cannot be served, which is the cheap place to fail – no GPU work has happened yet.

Parameters:: payload – Keyed by fully-qualified request field (“request:messages”). What the request supplies is derived from what it carries, so a topology cannot route around a modality the caller declared but did not bring. The storage behind each value is borrowed until take().

std::vector<Response> run()#

Drive every admitted request to completion, one response per seat.

Responses come back in admission order. Members share one step loop, so a stage that batches sees them together; submit() caps the cohort at one on a route through a stage that does not. Each member’s abandon token is polled at every step. A member that is cancelled, whose prepare(), finish() or diagnostics() throws, or whose stage returns kFailed does not stop the others; an exception from stepCohort() fails every live member.

void cancel(RequestSlot &slot) noexcept#: Request cancellation. Safe from another thread; the step loop observes it at the next abandon point. Reaches EVERY active stage, not just the first one.

void take()#: Reset every stage and release every seat, whether or not run() was called. A cohort that never ran gets no finish().

Public Static Functions

static void checkLoadable( MultiStageConfig const &config, StageRegistry const &runners )#: What load() checks of a validated config before it builds a stage: each entry against its runner’s declaration in runners, and each edge against the lanes. Touches no device.

struct Sinks#

Where a caller collects a terminal’s output.

Attaching a channel is what makes a terminal reachable, so this is also how “only text, no audio” is expressed: leave the audio pointer null and every stage that exists only to produce audio drops out of the route.

Public Functions

std::vector<std::string> attachedTerminals( MultiStageConfig const &config ) const#: Terminal port ids (“<stage>:<port>”) that have a channel attached. Empty when nothing is attached, which Route::resolve rejects.

Public Members

std::shared_ptr<StreamChannel> text#: Shared, not borrowed: the backbone holds this channel from prepare() to finish() and its abandon wake captures it, so its lifetime cannot be tied to the caller’s stack frame.

AudioStreamChannel *audio = {nullptr}#

VideoStreamChannel *video = {nullptr}#

struct Response#

Per-terminal outcome. A terminal that was cancelled or failed says so here; a caller cannot infer it from the channel alone.

Public Members

uint64_t requestId = {0}#

std::vector<std::pair<std::string, FinishReason>> terminalReasons#

bool ok = {false}#: No stage failed. A cancelled request is ok; terminalReasons says cancelled.

std::string error#

nlohmann::json stageDiagnostics = nlohmann::json::object()#: Per-stage report, keyed by stage id. Collected after finish(), before reset(), so a caller sees it whether or not the request succeeded. A stage whose diagnostics() throws has no entry, and the request fails.