DSparkDecodingConfig

class tensorrt_llm.llmapi.DSparkDecodingConfig(

*,

max_draft_len: Annotated[int, Ge(ge=0)] | None = None,

max_total_draft_tokens: int | None = None,

speculative_model: str | Path | None = None,

moe_backend: Literal['AUTO', 'CUTLASS', 'CUTEDSL', 'CUTEDSL_FC12', 'TRTLLM', 'DEEPGEMM', 'DENSEGEMM', 'VANILLA', 'TRITON', 'MARLIN', 'MEGAMOE_DEEPGEMM', 'MEGAMOE_CUTEDSL'] | None = None,

max_concurrency: Annotated[int, Gt(gt=0)] | None = None,

draft_len_schedule: dict[int, int] | None = None,

load_format: str | None = None,

acceptance_rate_window_size: Annotated[int, Ge(ge=0)] | None = None,

acceptance_rate_threshold: Annotated[float | None, Ge(ge=0.0), Le(le=1.0)] = None,

use_rejection_sampling: bool = False,

allow_advanced_sampling: bool = False,

advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL,

enable_penalty: bool = False,

mask_token_id: int | None = None,

target_layer_ids: List[int] | None = None,

block_size: Annotated[int, Gt(gt=0)] | None = None,

markov_rank: int | None = None,

markov_head_type: Literal['vanilla', 'gated', 'rnn'] | None = None,

decoding_type: Literal['DSpark'] = 'DSpark',

attention_backend: Literal['AUTO', 'VANILLA', 'TRTLLM', 'CUTEDSL'] = 'AUTO',

)

Bases: DecodingBaseConfig

Configuration for DSpark speculative decoding.

DSpark (DeepSeek) is a target-dependent, “semi-parallel” speculative decoding method. Like DFlash it captures hidden states from several target layers as cross-attention context and drafts a whole block in a single backbone forward, but it additionally refines the per-position draft logits with a lightweight sequential head (a low-rank Markov head, optionally an RNN head) and predicts an acceptance-confidence per position to truncate the proposed prefix.

Key features: - Target-dependent: captures hidden states from target_layer_ids. - Semi-parallel: one block backbone forward + cheap sequential head refine. - Confidence head: truncates the proposed draft length (NOT the accept rule;

acceptance stays standard target verification, preserving greedy parity).

Reference: DeepSeek DeepSpec (deepseek-ai/DeepSpec).

field acceptance_rate_threshold: float | None = None

The threshold for average true acceptance rate (accepted_draft_tokens / drafted_tokens); speculation will be disabled permanently once the rolling average over the last N speculation-enabled decoding iterations (N = acceptance_rate_window_size) drops below this value.

Constraints:

• ge = 0.0

• le = 1.0

field acceptance_rate_window_size: NonNegativeInt | None = None

The rolling average window size (N) for acceptance rate across speculation-enabled decoding iterations. If not set or set to 0, the feature is disabled. PyTorch backend only.

field advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL

Deploy-time specialization of the one-model advanced sampler that skips disabled filter kernels. FULL (default): per-row top_k/top_p/min_p in one fused kernel, the only mode accepting min_p. NO_TOPK: skip top_k. NO_TOPP: skip top_p. NO_TOPK_NO_TOPP: skip both.

field allow_advanced_sampling: bool = False

DEPRECATED: no-op kept for backward compatibility. Will be removed in a future release. Non-greedy sampling is now auto-detected per request; this flag no longer has any effect.

field attention_backend: Literal['AUTO', 'VANILLA', 'TRTLLM', 'CUTEDSL'] = 'AUTO'

Block-decode attention backend for a standalone DSpark drafter (one shipped as its own checkpoint, not inside the target’s mtp.* namespace). Ignored by the embedded DeepSeek-V4-Pro draft. Independent of the backend that builds the drafter’s own attention modules.

AUTO resolves per drafter family and is right unless you are pinning a kernel: a GQA backbone degrades when its kernel is missing, an MLA one raises. TRTLLM needs FlashInfer and SM100/SM103. CUTEDSL is MLA-only and needs a cute-dsl MLA decode taking per-token kv_bounds that is not upstream yet. Which kernel each name selects: MLADSparkForCausalLM.

field block_size: Annotated[int, Gt(gt=0)] | None = None

Number of draft positions produced per block. If None, read from the draft model config (dspark_block_size). Should equal max_draft_len.

field decoding_type: Literal['DSpark'] = 'DSpark'

field draft_len_schedule: dict[int, int] | None = None

Developer interface: dynamically adjust draft length based on active batch size in runtime.Maps batch size to draft lengths.For example: draft_len_schedule = {4:4, 8:2, 32:1} - Batch sizes 1-4: use draft_len=4 - Batch sizes 5-8: use draft_len=2 - Batch sizes 9-32: use draft_len=1 - Batch sizes 33+: use draft_len=0 (implicit, speculation disabled). Mutually exclusive with max_concurrency since draft_len_schedule implicitly support max concurrency control.

field enable_penalty: bool = False

If true, enables the occurrence penalties (repetition / presence / frequency) for one-model speculative decoding. Off by default because the penalties need a [num_seq_slots, vocab_size] occurrence-count workspace that is allocated up front (CUDA graphs capture fixed buffer addresses). While off, a request that asks for any of these penalties is rejected at admission rather than silently decoded without them.

field load_format: str | None = None

The load format of the speculative model.

field markov_head_type: Literal['vanilla', 'gated', 'rnn'] | None = None

Type of the sequential refinement head used within a block. If None, read from the draft model config (dspark_markov_head_type), defaulting to “vanilla”.

field markov_rank: int | None = None

Low-rank dimension of the Markov head logit-bias. If None, read from the draft model config (dspark_markov_rank). 0 disables the head.

field mask_token_id: int | None = None

Token ID used as the mask/noise token for parallel draft prediction. If None, read from the draft model config (dspark_noise_token_id).

field max_concurrency: PositiveInt | None = None

When specified (>0), speculation will be disabled at batch sizes above this value. Otherwise, speculation will always be on. PyTorch backend only. Mutually exclusive with max_concurrency since draft_len_schedule implicitly supports max concurrency control.

field max_draft_len: NonNegativeInt | None = None

The maximum number of draft tokens.

field max_total_draft_tokens: int | None = None

The number of draft tokens in the draft tokens tree. If it’s a linear tree, each draft layer will only generate one draft token. In this case, max_draft_len == max_total_draft_tokens. If it’s a static or dynamic tree, each draft layer may generate more than one draft token. In this case, max_total_draft_tokens >= max_draft_len.

field moe_backend: _MoeBackend | None = None

MoE backend override for a neural draft model or embedded MTP layers on the PyTorch backend. None inherits the target model’s backend. AUTO resolves from the draft checkpoint or embedded MTP layer quantization, and a concrete backend applies only to the draft model or layers. Resolution may fall back based on model, quantization, and hardware support. Replacement-head MTP checkpoints are unsupported because their independent quantization metadata is not loaded. Nemotron-H embedded MTP layers must inherit the target backend because their checkpoint mapper uses a shared backend-dependent layout. Decoding methods without a neural draft model ignore this option.

field speculative_model: str | Path | None = None

The speculative (draft) model. Accepts either (1) a HuggingFace Hub model ID (e.g. ‘yuhuili/EAGLE3-LLaMA3.1-Instruct-8B’), which will be automatically downloaded, or (2) a local filesystem path to a downloaded model directory. For one-model MTP, a non-target checkpoint provides either replacement MTP heads or a complete external draft model, depending on the target model implementation. Pointing it at the target checkpoint uses the target’s embedded mtp.* weights.

field target_layer_ids: List[int] | None = None

Target model layer indices whose hidden states are captured for cross-attention in the draft model. If None, read from the draft model config (dspark_target_layer_ids).

field use_rejection_sampling: bool = False

If true, enables rejection sampling for one-model speculative decoding paths when the batch contains any non-greedy request. All-greedy batches always take the argmax fast path regardless of this flag. Set to false (default) to use exact-match verification on non-greedy batches. The non-dynamic-tree one-model path requires FlashInfer.

__init__(**data: Any) → None

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

get_runtime_tokens_per_gen_step(

runtime_draft_len: int,

) → int

Total tokens per gen request for the current runtime draft length.

num_capture_layers() → int

validator set_max_total_draft_tokens  »  all fields

supports_backend(backend: str) → bool

Override if the speculation algorithm does not support a subset of the possible backends.

validator validate_draft_len_schedule_and_sort  »  draft_len_schedule

Validate and sort draft_len_schedule by batch size thresholds.

validator validate_max_concurrency_and_draft_len_schedule_mutually_exclusive  »  all fields

validator validate_rejection_sampling_config  »  all fields

Disable rejection sampling when SA-enhanced configurations are active.

Only silently disable a default-inherited value; an explicit use_rejection_sampling=True is preserved so TorchLlmArgs.validate_speculative_config can raise for the unsupported SA combination.

property draft_is_embedded_in_target: bool

True for the embedded (DeepSeek-V4-Pro) flavour of the DSpark draft.

DSpark ships in two shapes, and they need different runtime plumbing:

• embedded: the draft is the mtp.* namespace of the target checkpoint, built from full target decoder blocks, and served by DSv4DSparkWorker with its own rolling captured-context window.

• standalone: the draft is its own checkpoint with a registry-resolved backbone, served by DFlashWorker and its paged draft KV cache.

Both are decoding_type: DSpark, so every dispatch that must tell them apart – draft-model builder, worker, spec metadata, and the separate-draft-KV-cache decision – reads this one flag instead of re-deriving it. That is what keeps those decisions from drifting apart: a builder and a worker that disagree produce a draft model whose attributes the worker does not have.

The probe is the weight index rather than a config field because the index is authoritative and cannot be left unset; model_type is the fallback for a checkpoint whose index file is absent. Resolution is memoized here and warmed during TorchLlmArgs validation, so the filesystem probe happens once in the main process – not per rank, and never at forward or CUDA-graph-capture time.

property is_linear_tree: bool

property needs_separate_draft_weights: bool

Whether draft weights must be loaded from speculative_model.

This includes external draft models and MTP head replacement checkpoints.

property spec_dec_mode

property tokens_per_gen_step: int

DSpark needs K+1 tokens per gen request (K drafts + 1 bonus).

The draft produces its own mask queries internally; passing mask fillers through the target is pure wasted work at large batch size.

property uses_external_draft_model: bool

Whether speculative_model contains an external draft model.

property uses_replacement_heads: bool

Whether speculative_model contains replacement MTP heads.