Streaming#

class StreamChannel#

Per-slot streaming channel.

Encapsulated MPSC pipe between the runtime (producer) and a single consumer. State is private; consumer interacts via consume()/waitPop()/tryPop()/cancel(). Runtime-side operations (push, finish, setOriginalBatchIdx) are accessed via friendship so consumers cannot hold the lock or skip chunks.

Public Functions

inline bool getCollectMetrics() const noexcept#

Collect per-request wall timings without enabling aggregate GPU profiling.

StreamChannel(StreamChannel const&) = delete#
StreamChannel &operator=(StreamChannel const&) = delete#
StreamChannel(StreamChannel&&) = delete#
StreamChannel &operator=(StreamChannel&&) = delete#
~StreamChannel() noexcept = default#
template<typename Handler>
void consume(
Handler &&handler,
std::chrono::milliseconds poll = std::chrono::milliseconds{100}
)#

Block until finished or cancelled, delivering every chunk.

Exits when pending is empty AND either finished or cancelled is set. Lock is held only for pop; handler runs with the mutex released.

Template Parameters:

Handler – Callable with signature void(StreamChunk&&).

Parameters:
  • handler – Invoked once per chunk delivered.

  • poll – Max interval between wakeups (defaults to 100ms).

std::optional<StreamChunk> tryPop()#

Non-blocking single-chunk pop.

std::optional<StreamChunk> waitPop(
std::chrono::milliseconds timeout
)#

Blocking single-chunk pop with timeout.

Returns std::nullopt on timeout, or when woken by finish()/cancel() with empty deque. Callers that need the terminal signal should combine waitPop() with isFinished()/isCancelled().

bool isFinished() const noexcept#
FinishReason getReason() const noexcept#
int32_t getOriginalBatchIdx() const noexcept#
bool isCancelled() const noexcept#
void cancel() noexcept#

Fire-and-forget cancellation. Safe from any thread. Wakes blocked consume/waitPop.

void setStreamInterval(int32_t n) noexcept#

Per-request emit throttle. Values < 1 are clamped to 1. Default 1.

int32_t getStreamInterval() const noexcept#
void setSkipSpecialTokens(bool skip) noexcept#

Filter special tokens (EOS, <|im_end|>, <think>, vision placeholders, etc.) out of chunk.text. Default true. chunk.tokenIds is unaffected either way. Set before the channel is attached.

bool getSkipSpecialTokens() const noexcept#

Public Static Functions

static std::shared_ptr<StreamChannel> create(
bool collectMetrics = false
)#

Factory — only way to construct. Enforces shared_ptr ownership.

Friends

friend class scheduler::RequestEngine
template<typename Chunk>
class MediaStreamChannel#

Thread-safe MPSC queue carrying media chunks from a pipeline stage to a single consumer. push()/finish() are public (unlike StreamChannel’s friend-guarded producer API) because producers are pipeline callbacks composed outside the runtime.

Public Functions

inline std::optional<Chunk> waitPop(
std::chrono::milliseconds timeout
)#

Blocking pop; nullopt on timeout or terminal wake — combine with isFinished()/isCancelled(). A zero timeout is a non-blocking pop.

inline bool isFinished() const noexcept#
inline bool isCancelled() const noexcept#
inline void cancel() noexcept#

Wakes a blocked waitPop; producers observe it via isCancelled().

inline void push(Chunk chunk)#
inline void finish() noexcept#

Idempotent end-of-stream marker.

class AudioStreamChannel : public trt_edgellm::rt::MediaStreamChannel<AudioChunk>#
class VideoStreamChannel : public trt_edgellm::rt::MediaStreamChannel<VideoChunk>#
class StreamChannelFinalizer#

RAII guard that guarantees every attached StreamChannel terminates.

On destruction, for every slot whose channel has not yet been finalized the guard:

  1. Tries to build a terminal chunk containing any un-emitted tokens + text (sanitized + flushed to U+FFFD) with reason=kError and push()es it.

  2. Calls finish(kError) — idempotent, no-throw — to guarantee the consumer unblocks even if (1) failed under OOM.

Public Functions

StreamChannelFinalizer(
DecodingInferenceContext &ctx,
tokenizer::Tokenizer const &tok
) noexcept#
~StreamChannelFinalizer() noexcept#
StreamChannelFinalizer(StreamChannelFinalizer const&) = delete#
StreamChannelFinalizer &operator=(
StreamChannelFinalizer const&
) = delete#
StreamChannelFinalizer(StreamChannelFinalizer&&) = delete#
StreamChannelFinalizer &operator=(StreamChannelFinalizer&&) = delete#
struct StreamChunk#

Single delta chunk on a streaming channel.

text is always well-formed UTF-8 when pushed by the runtime — invalid byte sequences are replaced with U+FFFD before emission. Empty text chunks are legal (e.g., terminal chunks after all bytes were emitted).

Public Members

std::vector<int32_t> tokenIds#

Delta tokens since last chunk (may be >1 under spec-decode).

std::string text#

Delta text; always well-formed UTF-8.

int32_t promptTokenCount = {-1}#

Exact input-token count, or -1 when unavailable.

StreamMetrics metrics#

Producer-side cumulative snapshot, independent of emission interval.

bool finished = {false}#

True for the final chunk on this channel.

FinishReason reason = {FinishReason::kNotFinished}#

Terminal reason (only meaningful when finished==true).

std::vector<std::vector<LogprobEntry>> logprobs#

Per-token top-K logprobs: logprobs[i] = [LogprobEntry, …] for tokenIds[i]. Empty when numLogprobs == 0.

int32_t reusedPromptTokenCount = {-1}#

Prompt tokens served from reused state rather than prefilled, or -1 before the prefill has run.

struct StopMatchOutcome#

Result of applyStopStringMatch.

Public Members

std::string emitted#

Bytes safe to push right now.

bool stopMatched = {false}#

True iff a stop string was found and triggered truncation.

struct AudioChunk#

One PCM chunk emitted by a streaming audio (TTS / Omni) pipeline.

Public Members

std::string pcm16#

Little-endian int16 PCM samples, interleaved across channels.

bool isFinal = {false}#

True on the last chunk of the request.

int32_t numFrames = {0}#

Codec frames vocoded into this chunk.

int32_t sampleRate = {0}#

Samples per second of each channel.

int32_t numChannels = {1}#

Interleaved channels in pcm16.

struct VideoChunk#

One group of decoded frames emitted by a video generation pipeline.

Public Members

std::string rgb8#

Packed [numFrames, height, width, 3] 8-bit RGB.

bool isFinal = {false}#

True on the last chunk of the request.

int32_t numFrames = {0}#
int32_t height = {0}#
int32_t width = {0}#
int32_t fps = {0}#
struct SlotStreamState#

Per-slot detokenization and streaming state.

Lives in DecodingInferenceContext, compacted in lockstep with tokenIds.

Public Members

std::shared_ptr<StreamChannel> channel#

Channel (null ⇒ streaming disabled for this slot).

std::shared_ptr<std::atomic<bool> const> cancelToken#

The request’s own cancellation bit, when its owner supplied one (the request engine does). Polled at the top of every step beside the channel’s cancel flag; either stops the slot.

bool skipSpecialTokens = {true}#

Non-streaming detokenization policy. Streaming slots read this from their channel.

metrics::LLMRequestProfiler profiler#
size_t sentTokenCount = {0}#

Count of tokens whose piece bytes have been fed through emitDelta.

size_t lastEmittedTokenCount = {0}#

sentTokenCount at last push (for streamInterval gating).

size_t logprobsEmittedCount = {0}#

Number of logprob steps already pushed (0-based, independent of prompt length).

std::string pendingBytes#

Trailing incomplete UTF-8 bytes carried across iterations.

FinishReason terminalReason = {FinishReason::kNotFinished}#

Terminal reason latched at the moment finishedStates[i] flips to 1. One writer (the code that flips the state), one reader (the emit hook).

std::string stopMatchBuffer#

Hold-back buffer for cross-iteration stop-string matching; bounded by maxStopLen - 1 bytes.

size_t maxStopLen = {0}#

Cached max(stopStrings[*].size()) for this slot; 0 if no stops. Pre-computed once at handleRequest entry so applyStopStringMatch can skip the per-call scan.

std::string pendingEmitText#

Per-iteration emit text; populated by decodePerSlot, consumed (moved out) by emitChunks. Empty between iterations.

bool stopMatchedThisIter = {false}#

True iff stop-string match fired in decodePerSlot this iteration. Read by applyStopMatches to apply the kStopWords override; reset at top of each decodePerSlot pass.

void trt_edgellm::rt::attachStreamChannel(
std::shared_ptr<StreamChannel> const &channel,
int32_t originalIdx
)#
void trt_edgellm::rt::detachStreamChannel(
std::shared_ptr<StreamChannel> const &channel
)#

Undo attachStreamChannel for a channel nothing was pushed to, so a later admission can attach it again.

bool trt_edgellm::rt::validateStreamingSubmission(
LLMGenerationRequest const &request
)#
char const *trt_edgellm::rt::finishReasonName(
FinishReason r
) noexcept#

Human-readable name for a FinishReason. Returns a static, NUL-terminated C-string. Useful for logs, footer lines, and JSON output.

StopMatchOutcome trt_edgellm::rt::applyStopStringMatch(
std::string &buffer,
std::vector<std::string> const &stops,
size_t maxStopLen,
bool isFinal
)#

Stop-string match against a per-slot rolling buffer.

Caller appends new decoded bytes into buffer before invoking and passes maxStopLen = max(stops[*].size()) (cached at request entry; 0 ⇒ no stops or all-empty entries — degenerates to pass-through). On return:

  • Match found (earliest position wins): emitted = bytes before the match, stopMatched = true, buffer cleared.

  • isFinal, no match: full flush — emitted = buffer, buffer cleared.

  • Otherwise: emit buffer minus its trailing maxStopLen - 1 bytes; those bytes stay in buffer for cross-iteration matching.

Matching is byte-level; caller must guarantee UTF-8.

void trt_edgellm::rt::applyCancellationToFinishState(
DecodingInferenceContext &context,
int32_t row
)#

The one-row forms of the three stages above (row is an execution row of the context, not a resident slot); the batch forms are loops over them. A token produced outside a batch-wide step – the lookahead token of a sequence’s seated prefill – takes exactly the same path through these, so there is one pipeline for every produced token rather than a per-step one and a per-admission one that drift apart.

void trt_edgellm::rt::applyCancellationToFinishStates(
DecodingInferenceContext &context
)#
void trt_edgellm::rt::decodeSlot(
DecodingInferenceContext &context,
int32_t row,
tokenizer::Tokenizer const &tok
)#
void trt_edgellm::rt::decodePerSlot(
DecodingInferenceContext &context,
tokenizer::Tokenizer const &tok
)#

Stage 1 of the per-iter pipeline: decode new tokens into per-slot UTF-8 bytes and run stop-string matching. Sets s.pendingEmitText (bytes safe to emit this iter) and s.stopMatchedThisIter (signal for applyStopMatches to apply the kStopWords override). Does not push chunks and does not modify finishedStates / terminalReason — termination decisions live in updateFinishStates and applyStopMatches.

void trt_edgellm::rt::emitChunk(
DecodingInferenceContext &context,
int32_t row,
tokenizer::Tokenizer const &tokenizer
)#
std::vector<LogprobEntry> trt_edgellm::rt::logprobEntriesForStep(
LogprobsSlot const &slot,
int32_t step,
int32_t topK,
tokenizer::Tokenizer const &tokenizer
)#

The top-K entries of one generated token, with their pieces. A row written for a grammar-forced token holds only that token (logprob 0) followed by filler entries with a negative id, which are left out.

void trt_edgellm::rt::emitChunks(
DecodingInferenceContext &context,
tokenizer::Tokenizer const &tokenizer
)#

Stage 2 of the per-iter pipeline (after applyStopMatches): push chunks to channels using the pre-computed pendingEmitText and finalized terminalReason. Non-streaming slots are skipped here — their output is assembled at handleRequest finalization. tokenizer is used to resolve the raw token piece for per-token logprobs (LogprobEntry::piece).

void trt_edgellm::rt::emitTokenCallbackForSlot(
DecodingInferenceContext &context,
int32_t row
)#

Fires context.onTokenGenerated for every token row appended since its previous callback emission. The seated prefill runs it for its one slot, so a sequence’s first token reaches the callback the way every later one does.

void trt_edgellm::rt::emitTokenCallbacks(
DecodingInferenceContext &context
)#

The batch form: every slot, in slot order. Speculative decoders and DiffusionGemma can accept more than one token in a single decode iteration.