Runtime Module#
API documentation for the runtime module.
- Audio Loader
- Audio Utils
- Block Diffusion Decoder
- Block Hash
- Block Index
- Bounded Swa KV Page Manager
- Cache Record
- Checkpoint Reader
- Checkpoint Weight Assemble
- Collective Group
- Compact Capability
- Compact Media Reservation
- Context Cache Config
- Context Cache Coordinator
- Context Cache Deployment
- Context Cache Manager
- Context Cache Metrics
- Context Cache Request
- Context Cache Types
- Dart Pruner
- Decoder Registry
- Decoder Utils
- Decoding Inference Context
- Decoding Strategy
- Deepstack Binding
- Deployment Config
- Deployment Options
- Device Failure
- Dflash Decode Utils
- Dflash Decoder
- Domino Decoder
- Dspark Decoder
- EAGLE Decoder
- Embedding Preprocessor
- Encoder Embedding Cache
- Engine Executor
- Eviction Planner
- External Weight Manager
- Gdn Conv Commit Policy
- Gemma4 Embedding Preprocessor
- Gemma4 Mtp Decoder
- Guided Decoder
- Hybrid Cache Manager
- Hybrid Snapshot Storage
- Image Utils
- Inference Dims
- Inference Phase
- Jump Forward
- KV Cache Manager
- KV Page Table
- Layer Debugger
- LLM Engine Config
- LLM Inference Runtime
- LLM Rank Runtime
- LLM Runner
- LLM Runtime Utils
- Logit Bias
- Lora Manager
- Mamba2 Commit Policy
- Mamba Cache Manager
- Managed KV Cache Request
- Mel Spectrogram
- Model Artifacts
- Mtp Decoder
- Multi Device Plugin Comm Registry
- Multi Device Plugin Resource Factory
- Multi Device Plugin Resources
- Multi Stage Runtime
- Nccl Collective Backend
- Ngram Row Ids
- Parallel Config
- Pipeline Io
- Ple Cache
- Ple File Utils
- Ple Io Budget
- Ple Read Executor
- Ple Read Plan
- Ple Row Store
- Ple Runtime Config
- Ple Table
- Qwen3 Omni Tts Runtime
- Qwen4 Exp Ngram Preprocessor
- Ragged Batch Builder
- Recurrent Commit Policy
- Recurrent Commit Policy Factory
- Recurrent State Manager
- Recurrent State Types
- Refit Weights
- Registry Builder
- Request Stable Rng
- Resident Slot Pool
- Resource Pools
- Reuse Plan
- Rope Cache
- Route
- Rtx Runtime Cache
- Runtime Benchmark Options
- Runtime Coordinator
- Runtime Resources
- Runtime Session
- Scheduled Step
- Session Options
- Shared Resources
- Spec Common State Tracker
- Spec Decode Utils
- Spec Ple Staging
- Spec Reuse Contract
- Spec State Plan
- Stage
- Stage Config
- Stage Registry
- Step Preparer
- Streaming
- Swa KV Cache State
- Tensor Map
- Tensor Parallel Nccl Resources
- Tensor Parallel Plugin Resources
- Tensor Registry
- Vanilla Decoder
- Vision Gather
- Visual Token Pruner