Qwen3 Omni Tts Runtime#

class Qwen3OmniTTSRuntime#

Talker runtime for Qwen3-Omni RVQ code generation.

LLM-based codec encoder that generates RVQ codes from text tokens and hidden states. Manages two LLM engines (Talker + CodePredictor) and MLP projection layers.

Pipeline:

MLP Projection: thinker embed (layer 0) → talker embeddings via text_projection
Talker LLM: generate codec tokens autoregressively
CodePredictor: generate multi-layer codebook codes (Omni: 15, TTS: 31)
Return RVQ codes (vocoding done separately at example layer)

Architecture Philosophy:

Talker is an LLM decoder, NOT a multimodal input encoder
Similar to LLMInferenceRuntime, manages multiple LLM engines
Standalone runtime, not dependent on MultimodalRunner hierarchy
Code2Wav vocoding is separated for better modularity

Public Types

using AudioChunkCallback = std::function<void(std::vector<std::vector<int32_t>> const &chunkRvqCodes, bool isFinal)>#

Configuration for Thinker→Talker streaming pipeline.

Callback contract matches the TTS streaming path (see TalkerGenerationRequest::onChunkReady): chunkRvqCodes: codes since the last callback ([frames][numCodesPerFrame]) isFinal: true on the last callback for this request (post-EOS / post-maxAudioLength flush); invoked exactly once with isFinal=true when streaming is enabled, even if the chunk is empty (acts as the end-of-stream signal).

Public Functions

Qwen3OmniTTSRuntime( std::string const &talkerEngineDir, std::string const &codePredictorEngineDir, std::string const &tokenizerDir, std::string const &cloneEncoderDir, cudaStream_t stream, std::string const &checkpointDir = "", int32_t cpSpecVerifySize = 0 )#

Construct and fully initialize the TTS runtime.

Parameters:

talkerEngineDir – Directory containing talker engine, MLP weights, embedding table, etc.
codePredictorEngineDir – Directory containing code_predictor engine and codec embeddings
tokenizerDir – Directory containing tokenizer files. If empty, defaults to talkerEngineDir/../
checkpointDir – HF/ModelOpt checkpoint used by checkpoint-backed Talker weights
stream – CUDA stream for operations
cloneEncoderDir – Optional directory with the voice-clone reference encoder engines (speaker_encoder.engine / speech_tokenizer_encoder.engine, Base checkpoints). Empty disables voice cloning.
cpSpecVerifySize – CodePredictor verify window: one committed RVQ depth plus cpSpecVerifySize - 1 drafted depths. 0 disables speculation and keeps the autoregressive loop; 1 is rejected because it leaves no draft slot. The runtime falls back to the autoregressive loop when the engine or checkpoint cannot support the requested width.

Throws:

std::runtime_error – on any initialization failure

~Qwen3OmniTTSRuntime()#: Destructor.

inline std::vector<int32_t> getThinkerHiddenLayerIndices() const#: Hidden-state layer indices the Talker reads from the Thinker portal. Layer 0 is the input embedding; the second index is the decoder layer whose pre-norm hidden_states the Talker consumes, sourced from the Talker config’s accept_hidden_layer field. Must match the layer the Thinker engine was exported to emit on its hidden_states output.

bool handleAudioGeneration( std::vector<TalkerGenerationRequest> const &requests, TalkerGenerationResponse &response, cudaStream_t stream )#

Generate audio with RVQ codes (batched)

Implements the complete nested generation loop for a batch of requests:

Talker generation loop (autoregressive, batched engine execution)
CodePredictor generation (mNumRvqLayers per Talker step, per-batch)
Residual connections
Sampling at Runtime Layer (batched)

This is the main entry point for audio generation, analogous to LLMInferenceRuntime::handleRequest() for standard LLM inference.

Note

Sampling parameters (temperature, topK, topP, repetitionPenalty) are taken from requests[0] and applied uniformly to all batches. This matches LLMInferenceRuntime’s design where SamplingParams is shared across the batch.

Parameters:

requests – Batch of requests, each containing per-batch input data
response – Response containing generated RVQ codes [batchSize][frames][codes]
stream – CUDA stream for execution

Returns:

True if generation succeeded, false otherwise

inline bool handleAudioGeneration( TalkerGenerationRequest const &request, TalkerGenerationResponse &response, cudaStream_t stream )#: Convenience wrapper for single-request audio generation.

bool handleAudioGenerationFromThinker( std::vector<OmniGenerationRequest> const &requests, TalkerGenerationResponse &response, cudaStream_t stream )#

Generate audio from external Thinker hidden states (Omni inference path, batched)

Instead of tokenizing text and looking up embeddings internally (TTS path), this API accepts pre-computed Thinker layer-0 hidden states and projects them through the MLP to produce Talker input. Used when integrating with llm_inference.

Note

Parameters:

requests – Batch of requests, each containing per-batch thinker embeddings
response – Response containing generated RVQ codes [batchSize][frames][codes]
stream – CUDA stream for execution

Returns:

True if generation succeeded, false otherwise

inline bool handleAudioGenerationFromThinker( OmniGenerationRequest const &request, TalkerGenerationResponse &response, cudaStream_t stream )#: Convenience wrapper for single-request Omni audio generation.

bool handleStreamingGeneration( LLMInferenceRuntime &thinkerRuntime, LLMGenerationRequest &thinkerRequest, LLMGenerationResponse &thinkerResponse, ThinkerTalkerStreamingConfig const &streamingConfig, OmniGenerationRequest const &omniBaseRequest, TalkerGenerationResponse &talkerResponse, cudaStream_t stream )#

Streaming generation: Thinker and Talker run interleaved on the same CUDA stream.

Uses LLMGenerationRequest::onTokenGenerated to receive per-token callbacks from the Thinker’s decode loop. When enough assistant tokens accumulate, Talker prefill is triggered. Subsequent Thinker tokens incrementally extend trailing_text_hidden, and Talker decode steps are interleaved.

Parameters:

thinkerRuntime – Thinker LLM runtime (will call handleRequest internally)
thinkerRequest – Thinker request (onTokenGenerated will be overwritten)
streamingConfig – Pipeline tuning parameters
talkerResponse – Output: generated RVQ codes
stream – CUDA stream (shared by Thinker and Talker)

Returns:

True if the full pipeline succeeded

inline metrics::MultimodalMetrics const &getMetrics() const#

Get performance metrics for Talker pipeline (legacy, for backward compat)

Returns:: Reference to metrics object

inline metrics::OmniTalkerMetrics const &getOmniTalkerMetrics( ) const#: Get Omni-specific Talker metrics (frames, RVQ codes, prefill time, exit reason)

inline metrics::OmniLatencyMetrics const &getOmniLatencyMetrics( ) const#: Get Omni audio latency metrics (TTFA, RTF, E2E)

inline metrics::OmniLatencyMetrics &getMutableOmniLatencyMetrics()#: Get mutable reference to latency metrics (for E2E timing set at example layer)

inline cudaEvent_t getTtfaEndEvent() const#: Get the TTFA end event (first codec token sampled) for external timing.

bool captureDecodingCUDAGraph(cudaStream_t stream)#

Capture CUDA graphs for decoding steps (same pattern as LLMInferenceRuntime).

Parameters:: stream – CUDA stream for capture
Returns:: True if all graphs captured successfully

int32_t getSpeakerIdByName(std::string const &speakerName) const#

Get speaker ID by name.

Parameters:: speakerName – Speaker name (e.g., “f245”, “m02”)
Returns:: Speaker ID, or default speaker ID if not found

int32_t resolveLanguageId( std::string const &languageName, std::string const &speakerName ) const#

Resolve the language codec token ID for a request (CustomVoice language conditioning)

Mirrors the PyTorch reference (modeling_qwen3_tts.py):

Empty / “auto” language → -1 (no-language path), unless step 3 overrides.
Known language name (case-insensitive lookup in codecLanguageIdMap) → its codec ID. Unknown names log a warning and fall back to -1.
Dialect override: when the resolved language is “auto” or “chinese” and the speaker is a dialect speaker (spkDialectMap), the dialect’s codec ID wins.

Always returns -1 when the engine config has no codec_think_id / codec_language_id (e.g. Qwen3-Omni checkpoints or engines exported before language support).

Parameters:

languageName – Request language name (”” = auto)
speakerName – Request speaker name (for dialect override; “” = default speaker)

Returns:

Language codec token ID, or -1 for the no-language 8-row prefill

inline std::vector<std::string> getSpeakerNames() const#: Speaker names accepted by TalkerGenerationRequest::speakerName.

struct OmniGenerationRequest#

Request structure for Omni inference (Thinker output as input)

Non-streaming: provide fullText (formatted prompt + generated text), which will be tokenized internally to reconstruct layer-0 embeddings via the Thinker embedding table.

Public Members

std::string fullText#: Complete formatted text (if textTokenIds empty, tokenized internally)

std::vector<int32_t> textTokenIds#

Full token sequence: inputTokenIds + outputIds (including EOS)

Non-owning pointer to this batch’s prefill layer-0 embeddings (with multimodal features). Must point to a [1, prefillLength, thinkerHiddenSize] FP16 (GPU) view for this batch. Caller slices from the full [BS, prefillLen, H] tensor. Generated token embeddings are reconstructed from the TTS embedding table internally.

rt::Tensor const *thinkerPrefillEmbeds = {nullptr}#: Non-owning pointer to this batch’s layer-14 hidden states (prefill only). Must point to a [1, prefillLength, thinkerHiddenSize] FP16 (GPU) view for this batch. Only user-segment multimodal token positions are read.

rt::Tensor const *thinkerHiddenStates = {nullptr}#

int32_t prefillLength = {0}#: Number of prefill tokens (layer0/layer14 cover [0, prefillLength))

int32_t maxAudioLength = {4096}#

float talkerTemperature = {0}#

int32_t talkerTopK = {0}#

float talkerTopP = {0}#

float repetitionPenalty = {1.05f}#

std::string speakerName = {""}#

int32_t speakerId = {-1}#

struct TalkerGenerationRequest#

Talker audio generation request structure.

Contains sampling parameters and input data for audio generation. Sampling parameters are provided per-request (not from config.json).

Public Members

int32_t maxAudioLength = {4096}#: Maximum number of audio codec tokens to generate.

float talkerTemperature = {0}#: Talker temperature (0 = default 0.9)

int32_t talkerTopK = {0}#: Talker top-K (0 = default 50)

float talkerTopP = {0}#: Talker top-P (0 = default 1.0)

float repetitionPenalty = {1.05f}#: Repetition penalty applied to seen codec tokens (1.0 = disabled)

float subtalkerTemperature = {0}#: CP (sub-talker) temperature (0 = arch default)

int32_t subtalkerTopK = {0}#: CP (sub-talker) top-K (0 = kCPSamplingTopK)

float subtalkerTopP = {0}#: CP (sub-talker) top-P (0 = arch default)

std::string speakerName = {""}#: Speaker name (e.g., “f245”, “m02”) - empty means use default.

int32_t speakerId = {-1}#: Speaker ID - if >= 0, overrides speakerName.

std::string languageName = {""}#: < [Qwen3-TTS] CustomVoice language conditioning (optional). Empty or “auto” keeps the historical no-language prefill. A known language name (e.g. “chinese”, “english”; matched case-insensitively against the engine config codec_language_id map) switches the Talker prefill to the 9-row language-conditioned layout. Unknown names fall back to no-language with a warning. Dialect speakers (spk_is_dialect in config) override this automatically when language is “auto” or “chinese”, matching the PyTorch reference. [Qwen3-TTS] CustomVoice/VoiceDesign instruction control (optional). Natural-language style instruction (e.g. “Speak in a whisper”). Wrapped as a user turn, projected through text_projection, and prepended to the Talker prefill. Empty = no instruction.

std::string instructText = {""}#: [Qwen3-TTS] Voice clone (Base checkpoints, optional): reference audio file (wav/mp3/flac). The reference encoders run on-device (requires cloneEncoderDir at construction). With refText set, ICL mode conditions on (transcript, codec codes) of the reference; without it, x-vector-only mode clones timbre alone.

std::string refAudioPath = {""}#

std::string refText = {""}#: Reference transcript (enables ICL mode)

std::vector<std::vector<int32_t>> promptSpeakerCodes#: [Qwen3-Omni Next] Custom voice: reference-voice codec codes, [frames][num_code_groups]. When non-empty, replaces the built-in speaker codec rows and skips the per-speaker system prompt (HF prompt_speaker_codes semantics).

std::string assistantInstruct = {""}#: [Qwen3-Omni Next] Style/emotion instruction name resolved via config talker_assistant_prompt_id_mapping (e.g. “cheerful”); empty = none.

std::string talkerLanguage = {""}#: [Qwen3-Omni Next] Target language resolved via config talker_language_id (e.g. “chinese”); empty or “auto” = model decides (codec_nothink path).

std::string systemInstruct = {""}#: [Qwen3-Omni Next] Free-text system instruction inserted after the system role trio (HF talker_system_instruct_ids); tokenized by the runtime. Empty = none.

std::vector<Message> messages#

bool applyChatTemplate = {true}#: Whether to apply chat template formatting.

bool addGenerationPrompt = {true}#: Whether to add generation prompt at the end.

bool enableThinking = {false}#

Whether to enable thinking mode.

Optional streaming: emit RVQ codes via callback every N frames. 0 = non-streaming (default); >0 enables streaming for this request.

int32_t streamingChunkFrames = {0}#: Streaming callback invoked from the Talker generation thread with: chunkRvqCodes: codes generated since the last callback ([frames][numCodesPerFrame]) isFinal: true on the last callback for this request (post-EOS / post-maxFrames flush); invoked exactly once with isFinal=true per request that has streaming enabled, even if chunkRvqCodes is empty (signals end-of-stream). Each request gets its own callback; per-batch streams are fully independent.

std::function<void(std::vector<std::vector<int32_t>> const &chunkRvqCodes, bool isFinal)> onChunkReady#

struct TalkerGenerationResponse#

Talker audio generation response structure.

Contains generated RVQ codes and metadata.

Public Members

std::vector<std::vector<std::vector<int32_t>>> batchRvqCodes#

std::vector<int32_t> numFramesPerSample#: Number of audio frames generated per batch sample.

bool success = {false}#: Whether generation succeeded.

struct ThinkerTalkerStreamingConfig#

Public Members

int32_t talkerPrefillThreshold = {4}#: Start Talker prefill after this many assistant tokens.

int32_t codecChunkFrames = {0}#: Emit chunk every N frames (0 = disabled)

AudioChunkCallback onAudioChunkReady#: Per-chunk callback (see AudioChunkCallback contract)

struct TalkerGenerationRequest

Talker audio generation request structure.

Contains sampling parameters and input data for audio generation. Sampling parameters are provided per-request (not from config.json).

Public Members

int32_t maxAudioLength = {4096}: Maximum number of audio codec tokens to generate.

float talkerTemperature = {0}: Talker temperature (0 = default 0.9)

int32_t talkerTopK = {0}: Talker top-K (0 = default 50)

float talkerTopP = {0}: Talker top-P (0 = default 1.0)

float repetitionPenalty = {1.05f}: Repetition penalty applied to seen codec tokens (1.0 = disabled)

float subtalkerTemperature = {0}: CP (sub-talker) temperature (0 = arch default)

int32_t subtalkerTopK = {0}: CP (sub-talker) top-K (0 = kCPSamplingTopK)

float subtalkerTopP = {0}: CP (sub-talker) top-P (0 = arch default)

std::string speakerName = {""}: Speaker name (e.g., “f245”, “m02”) - empty means use default.

int32_t speakerId = {-1}: Speaker ID - if >= 0, overrides speakerName.

std::string languageName = {""}: < [Qwen3-TTS] CustomVoice language conditioning (optional). Empty or “auto” keeps the historical no-language prefill. A known language name (e.g. “chinese”, “english”; matched case-insensitively against the engine config codec_language_id map) switches the Talker prefill to the 9-row language-conditioned layout. Unknown names fall back to no-language with a warning. Dialect speakers (spk_is_dialect in config) override this automatically when language is “auto” or “chinese”, matching the PyTorch reference. [Qwen3-TTS] CustomVoice/VoiceDesign instruction control (optional). Natural-language style instruction (e.g. “Speak in a whisper”). Wrapped as a user turn, projected through text_projection, and prepended to the Talker prefill. Empty = no instruction.

std::string instructText = {""}: [Qwen3-TTS] Voice clone (Base checkpoints, optional): reference audio file (wav/mp3/flac). The reference encoders run on-device (requires cloneEncoderDir at construction). With refText set, ICL mode conditions on (transcript, codec codes) of the reference; without it, x-vector-only mode clones timbre alone.

std::string refAudioPath = {""}

std::string refText = {""}: Reference transcript (enables ICL mode)

std::vector<std::vector<int32_t>> promptSpeakerCodes: [Qwen3-Omni Next] Custom voice: reference-voice codec codes, [frames][num_code_groups]. When non-empty, replaces the built-in speaker codec rows and skips the per-speaker system prompt (HF prompt_speaker_codes semantics).

std::string assistantInstruct = {""}: [Qwen3-Omni Next] Style/emotion instruction name resolved via config talker_assistant_prompt_id_mapping (e.g. “cheerful”); empty = none.

std::string talkerLanguage = {""}: [Qwen3-Omni Next] Target language resolved via config talker_language_id (e.g. “chinese”); empty or “auto” = model decides (codec_nothink path).

std::string systemInstruct = {""}: [Qwen3-Omni Next] Free-text system instruction inserted after the system role trio (HF talker_system_instruct_ids); tokenized by the runtime. Empty = none.

std::vector<Message> messages

bool applyChatTemplate = {true}: Whether to apply chat template formatting.

bool addGenerationPrompt = {true}: Whether to add generation prompt at the end.

bool enableThinking = {false}

Whether to enable thinking mode.

Optional streaming: emit RVQ codes via callback every N frames. 0 = non-streaming (default); >0 enables streaming for this request.

int32_t streamingChunkFrames = {0}: Streaming callback invoked from the Talker generation thread with: chunkRvqCodes: codes generated since the last callback ([frames][numCodesPerFrame]) isFinal: true on the last callback for this request (post-EOS / post-maxFrames flush); invoked exactly once with isFinal=true per request that has streaming enabled, even if chunkRvqCodes is empty (signals end-of-stream). Each request gets its own callback; per-batch streams are fully independent.

std::function<void(std::vector<std::vector<int32_t>> const &chunkRvqCodes, bool isFinal)> onChunkReady

struct TalkerGenerationResponse

Talker audio generation response structure.

Contains generated RVQ codes and metadata.

Public Members

std::vector<std::vector<std::vector<int32_t>>> batchRvqCodes

std::vector<int32_t> numFramesPerSample: Number of audio frames generated per batch sample.

bool success = {false}: Whether generation succeeded.

struct OmniGenerationRequest

Request structure for Omni inference (Thinker output as input)

Non-streaming: provide fullText (formatted prompt + generated text), which will be tokenized internally to reconstruct layer-0 embeddings via the Thinker embedding table.

Public Members

std::string fullText: Complete formatted text (if textTokenIds empty, tokenized internally)

std::vector<int32_t> textTokenIds

Full token sequence: inputTokenIds + outputIds (including EOS)

rt::Tensor const *thinkerPrefillEmbeds = {nullptr}: Non-owning pointer to this batch’s layer-14 hidden states (prefill only). Must point to a [1, prefillLength, thinkerHiddenSize] FP16 (GPU) view for this batch. Only user-segment multimodal token positions are read.

rt::Tensor const *thinkerHiddenStates = {nullptr}

int32_t prefillLength = {0}: Number of prefill tokens (layer0/layer14 cover [0, prefillLength))

int32_t maxAudioLength = {4096}

float talkerTemperature = {0}

int32_t talkerTopK = {0}

float talkerTopP = {0}

float repetitionPenalty = {1.05f}

std::string speakerName = {""}

int32_t speakerId = {-1}

struct ThinkerTalkerStreamingConfig

Public Members

int32_t talkerPrefillThreshold = {4}: Start Talker prefill after this many assistant tokens.

int32_t codecChunkFrames = {0}: Emit chunk every N frames (0 = disabled)

AudioChunkCallback onAudioChunkReady: Per-chunk callback (see AudioChunkCallback contract)