Talker Runner#

class TalkerRunner : public trt_edgellm::rt::multiStage::IStage #

Codec generation, behind IStage.

One step is one codec frame: the frame is appended to the codec port and published before the next step runs, so a cancel lands within a frame rather than at the end of the utterance. The port carries every frame produced so far, which is the shape the vocoder downstream walks with its own cursor.

The residual code predictor is a second engine inside this stage, not a stage of its own: it runs once per codec frame from within stepCohort(), on this stage’s stream, over buffers this stage owns.

Config keys: prefill_builder required, no default. “segments”, “chunked” or “tts”; see PrefillBuilder. code_predictor_artifact required, code predictor engine directory relative to artifact code_predictor_checkpoint optional, checkpoint-backed code predictor weights, relative to code_predictor_artifact tokenizer_dir directory holding the backbone embedding.safetensors; defaults to the parent of artifact. Every layout looks its tts tokens up in the tokenizer here, else in artifact, and refuses to load with neither; the segments and chunked layouts also look up their chat-template tokens, and segments its media placeholders. A talker with no backbone renders requests with the tokenizer and chat template of the first of the two that has both checkpoint optional, checkpoint-backed weights speaker default speaker name or id; a request may override it via voice assistant_instruct style/emotion name resolved through talker_assistant_prompt_id_mapping language tts layout language resolved through codec_language_id; a request overrides it with the language style field talker_language chunked layout language resolved through talker_language_id; a request overrides it with the talker_language style field max_audio_length default codec-frame cap; a request may raise it up to the KV room left after its prefill temperature / top_k / top_p / repetition_penalty talker sampling cp_temperature / cp_top_k / cp_top_p code predictor sampling cp_spec_verify_size optional, code predictor verify window; 0 (default) keeps the autoregressive RVQ loop clone_encoder_artifact optional, reference-encoder engine directory relative to artifact report_codes optional, false by default; true adds the request’s codec frames to its report as rvq_codes, one array of codes per frame

Public Functions

explicit TalkerRunner(multiStage::StageInitContext const &init)#

~TalkerRunner() override#

virtual void preallocate() override#: Init-time allocation. Every device buffer this stage will ever use is claimed here, sized by maxBatch x maxSeq. Nothing on the step path may allocate.

virtual bool admit( multiStage::StageExecContext const &ctx, std::string &reason ) override#

Per-request admission. Reject here – it costs nothing – rather than failing mid-step.

Returns:: false plus a reason when this stage cannot serve the request.

virtual void prepare(multiStage::StageExecContext &ctx) override#: Per-request setup, once the inputs this stage reads are published. A throw fails only this request, so what prepare() writes must be this member’s own or scratch that the next member’s prepare() rewrites.

virtual multiStage::StageStatus step( multiStage::StageExecContext &ctx ) override#

void stepCohort( std::vector<multiStage::StageExecContext*> const &members, std::vector<multiStage::StageStatus> &statuses ) override#

virtual void finish( multiStage::StageExecContext &ctx, multiStage::FinishReason reason ) override#: Called for every member on every active stage when run() ends: success, failure and cancellation alike. Emits the terminating chunk and reports metrics. A stage that only cleans up on success is a leak. take() without run() skips it; see reset().

virtual void reset() override#: Idempotent. Drops per-request state; anything longer-lived survives.

virtual nlohmann::json diagnostics( rt::multiStage::RequestSlot const &slot ) const override#

What this stage did on slot's request, for that caller’s report.

Read after finish() and before reset(). Keys are the stage’s own; the runtime nests them under the stage id and never interprets them. An empty object means the stage has nothing to say, which is the default. Taking the slot is what keeps a cohort’s members from all reporting the same numbers. A throw fails only that request, which keeps the other stages’ reports.

This is deliberately not gated on profiling: a caller needs to know why generation stopped even when no timing was collected.

std::unique_ptr<multiStage::IStage> trt_edgellm::rt::qwen3_omni::createTalkerRunner( multiStage::StageInitContext const &init )#

multiStage::RunnerDecl trt_edgellm::rt::qwen3_omni::talkerRunnerDeclaration( )#: What a talker stage offers a stage graph, and how request files spell what it reads.