Stage

class PortBindings

Public Functions

void bind(std::string const &name, PortValue value)

PortValue const *find(std::string const &name) const

nullptr when the input was skipped. Callers must handle it; there is no default.

PortValue const &at(std::string const &name) const

Throws naming the input. Use when the config guarantees presence (OnMissing::kFail).

void clear()

std::vector<std::string> names() const

Every bound name, in unspecified order.

class AbandonToken

Cancellation token. Polled at every abandon point; never checked by a raw bool.

Public Functions

inline void request() noexcept

inline bool requested() const noexcept

inline void reset() noexcept

inline void onRequest(std::function<void()> wake)

Run wake when this token is requested.

A stage blocked inside its own decode loop never reaches the next abandon point, so it registers the one call that unblocks it – for the backbone, cancelling the stream channel the decode loop already polls once per iteration. Registering after the fact still fires, because a cancel that arrived first must not be lost.

class IStage

One computation, behind a uniform interface.

A stage reaches other stages only through its ports. The bindings, slots and streams it borrows are owned by the executor; the sinks and SharedResources belong to the caller.

Subclassed by trt_edgellm::rt::LlmRunner, trt_edgellm::rt::minimax_h3::AudioDecoderStage, trt_edgellm::rt::minimax_h3::DenoiserStage, trt_edgellm::rt::minimax_h3::TextEncoderStage, trt_edgellm::rt::minimax_h3::VideoDecoderStage, trt_edgellm::rt::qwen3_omni::Code2WavStage, trt_edgellm::rt::qwen3_omni::TalkerRunner

Public Functions

virtual ~IStage() = default

virtual void preallocate() = 0

Init-time allocation. Every device buffer this stage will ever use is claimed here, sized by maxBatch x maxSeq. Nothing on the step path may allocate.

virtual bool admit(

StageExecContext const &ctx,

std::string &reason

) = 0

Per-request admission. Reject here – it costs nothing – rather than failing mid-step.

Returns:

false plus a reason when this stage cannot serve the request.

virtual void prepare(StageExecContext &ctx) = 0

Per-request setup, once the inputs this stage reads are published. A throw fails only this request, so what prepare() writes must be this member’s own or scratch that the next member’s prepare() rewrites.

virtual StageStatus step(StageExecContext &ctx) = 0

inline virtual void stepCohort(

std::vector<StageExecContext*> const &members,

std::vector<StageStatus> &statuses

)

One step for every cohort member the executor has ready.

statuses is index-aligned with members. The default serves a cohort of one and refuses anything wider, because a stage holding a single request’s state cannot interleave members – round-robin would have member B step on A’s buffers. A stage that batches overrides this: members is the only place the cohort is visible together, so it is the only place a batched call can be formed.

virtual void finish(StageExecContext &ctx, FinishReason reason) = 0

Called for every member on every active stage when run() ends: success, failure and cancellation alike. Emits the terminating chunk and reports metrics. A stage that only cleans up on success is a leak. take() without run() skips it; see reset().

virtual void reset() = 0

Idempotent. Drops per-request state; anything longer-lived survives.

inline virtual nlohmann::json diagnostics(

RequestSlot const &slot

) const

What this stage did on slot's request, for that caller’s report.

Read after finish() and before reset(). Keys are the stage’s own; the runtime nests them under the stage id and never interprets them. An empty object means the stage has nothing to say, which is the default. Taking the slot is what keeps a cohort’s members from all reporting the same numbers. A throw fails only that request, which keeps the other stages’ reports.

This is deliberately not gated on profiling: a caller needs to know why generation stopped even when no timing was collected.

struct PortValue

A non-owning view of a value on a port, in host or device memory.

Public Members

void *data = {nullptr}

std::vector<int64_t> shape

PortKind kind = {PortKind::kText}

int32_t elementBytes = {0}

bool complete = {false}

The producer will not extend this value. Until it is set, a consumer that has drained everything bound so far must report kBlocked rather than kDone: an empty remainder means “not yet”, and only the producer knows when it means “never”.

struct RequestSlot

One in-flight request.

Public Members

uint64_t requestId = {0}

int32_t batchSlot = {-1}

Batch row this request occupies within its cohort.

Route route

AbandonToken abandon

PortBindings payload

What the caller supplied, keyed by the fully-qualified request field (“request:messages”, “request:audio”). Borrowed: the caller’s storage must outlive the request. The executor binds from here into each stage that reads a request:* input, which is the only way such an input reaches a stage.

struct StageExecContext

What a stage is handed on every per-request call. Borrowed, except textSink, which is shared.

Public Members

RequestSlot *slot = {nullptr}

PortBindings const *in = {nullptr}

PortBindings *out = {nullptr}

cudaStream_t stream = {nullptr}

This stage’s lane stream.

std::shared_ptr<StreamChannel> textSink

Where a terminal’s output leaves the topology. Null unless this stage emits a terminal port of that kind AND the caller attached a channel to it. A stage still does not learn its position: it only learns whether it has an exit.

AudioStreamChannel *audioSink = {nullptr}

VideoStreamChannel *videoSink = {nullptr}

struct StageInitContext

What a stage is handed once, at construction.

Public Members

StageConfig const *config = {nullptr}

std::string artifactDir

Already resolved and checked against the config root.

SharedResources *shared = {nullptr}

cudaStream_t stream = {nullptr}

int32_t maxBatch = {1}

int32_t maxSeq = {0}

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::rt::multiStage::toString” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:

- char const *toString(FinishReason v)
- char const *toString(OnMissing v)
- char const *toString(PortKind v)
- char const *toString(StageKind v)
- char const *toString(StageStatus v)
- char const *toString(Transport v)

nlohmann::json trt_edgellm::rt::multiStage::jsonInput(

PortBindings const *in,

std::string const &name

)

Read a UTF-8 JSON object bound on an input.

Returns an empty object when the input is absent, which is the common case: per-request overrides are optional and a stage falls back to its deployment config. Throws naming the input when something IS bound but does not parse, because silently ignoring a malformed override would run the request with settings the caller did not ask for.