Runtime Module

API documentation for the runtime module.

• Audio Loader
• Audio Utils
• Block Diffusion Decoder
• Block Hash
• Block Index
• Cache Record
• Checkpoint Reader
• Checkpoint Weight Assemble
• Collective Group
• Context Cache Config
• Context Cache Coordinator
• Context Cache Deployment
• Context Cache Manager
• Context Cache Metrics
• Context Cache Request
• Context Cache Types
• Decoder Registry
• Decoder Utils
• Decoding Inference Context
• Decoding Strategy
• Deepstack Binding
• Deployment Config
• Dflash Decode Utils
• Dflash Decoder
• Dspark Decoder
• EAGLE Decoder
• Embedding Preprocessor
• Engine Executor
• Eviction Planner
• External Weight Manager
• Gemma4 Embedding Preprocessor
• Gemma4 Mtp Decoder
• Hybrid Cache Manager
• Hybrid Snapshot Storage
• Image Utils
• Inference Dims
• Inference Phase
• KV Cache Manager
• KV Page Table
• Layer Debugger
• LLM Engine Config
• LLM Inference Runtime
• LLM Runtime Utils
• Logit Bias
• Lora Manager
• Mamba Cache Manager
• Mel Spectrogram
• Mtp Decoder
• Multi Device Plugin Comm Registry
• Multi Device Plugin Resource Factory
• Multi Device Plugin Resources
• Nccl Collective Backend
• Parallel Config
• Pipeline Io
• Qwen3 Omni Tts Runtime
• Registry Builder
• Resource Pools
• Reuse Plan
• Rope Cache
• Shared Resources
• Step Preparer
• Streaming
• System Prompt KV Cache
• Tensor Map
• Tensor Parallel Nccl Resources
• Tensor Parallel Plugin Resources
• Tensor Registry
• Vanilla Decoder