DSparkDecodingConfig

class tensorrt_llm.llmapi.DSparkDecodingConfig(

*,

max_draft_len: Annotated[int, Ge(ge=0)] | None = None,

max_total_draft_tokens: int | None = None,

speculative_model: str | Path | None = None,

max_concurrency: Annotated[int, Gt(gt=0)] | None = None,

draft_len_schedule: dict[int, int] | None = None,

load_format: str | None = None,

acceptance_rate_window_size: Annotated[int, Ge(ge=0)] | None = None,

acceptance_rate_threshold: Annotated[float | None, Ge(ge=0.0), Le(le=1.0)] = None,

use_rejection_sampling: bool = False,

allow_advanced_sampling: bool = False,

advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL,

enable_penalty: bool = False,

mask_token_id: int | None = None,

target_layer_ids: List[int] | None = None,

block_size: Annotated[int, Gt(gt=0)] | None = None,

markov_rank: int | None = None,

markov_head_type: Literal['vanilla', 'gated', 'rnn'] | None = None,

decoding_type: Literal['DSpark'] = 'DSpark',

attention_backend: Literal['VANILLA', 'TRTLLM'] = 'VANILLA',

)

Bases: DecodingBaseConfig

Configuration for DSpark speculative decoding.

DSpark (DeepSeek) is a target-dependent, “semi-parallel” speculative decoding method. Like DFlash it captures hidden states from several target layers as cross-attention context and drafts a whole block in a single backbone forward, but it additionally refines the per-position draft logits with a lightweight sequential head (a low-rank Markov head, optionally an RNN head) and predicts an acceptance-confidence per position to truncate the proposed prefix.

Key features: - Target-dependent: captures hidden states from target_layer_ids. - Semi-parallel: one block backbone forward + cheap sequential head refine. - Confidence head: truncates the proposed draft length (NOT the accept rule;

acceptance stays standard target verification, preserving greedy parity).

Reference: DeepSeek DeepSpec (deepseek-ai/DeepSpec).

field acceptance_rate_threshold: float | None = None

The threshold for average true acceptance rate (accepted_draft_tokens / drafted_tokens); speculation will be disabled permanently once the rolling average over the last N speculation-enabled decoding iterations (N = acceptance_rate_window_size) drops below this value.

Constraints:

• ge = 0.0

• le = 1.0

field acceptance_rate_window_size: NonNegativeInt | None = None

The rolling average window size (N) for acceptance rate across speculation-enabled decoding iterations. If not set or set to 0, the feature is disabled. PyTorch backend only.

field advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL

Deploy-time specialization of the one-model advanced sampler that skips disabled filter kernels. FULL (default): per-row top_k/top_p. NO_TOPK: skip top_k. NO_TOPP: skip top_p. NO_TOPK_NO_TOPP: skip both.

field allow_advanced_sampling: bool = False

DEPRECATED: no-op kept for backward compatibility. Will be removed in a future release. Non-greedy sampling is now auto-detected per request; this flag no longer has any effect.

field attention_backend: Literal['VANILLA', 'TRTLLM'] = 'VANILLA'

Attention backend for the pooled-context cross-attention of a standalone DSpark drafter (one shipped as its own checkpoint rather than inside the target’s mtp.* namespace). Ignored by the embedded DeepSeek-V4-Pro draft, which uses its own captured-context attention. This is independent of the backend used to construct the drafter’s standard attention modules. TRTLLM requires FlashInfer and an NVIDIA Blackwell GPU with SM100 or SM103, and uses generated FMHA kernels with a private paged context cache; VANILLA uses FlashAttention with a contiguous cache.

field block_size: Annotated[int, Gt(gt=0)] | None = None

Number of draft positions produced per block. If None, read from the draft model config (dspark_block_size). Should equal max_draft_len.

field decoding_type: Literal['DSpark'] = 'DSpark'

field draft_len_schedule: dict[int, int] | None = None

Developer interface: dynamically adjust draft length based on active batch size in runtime.Maps batch size to draft lengths.For example: draft_len_schedule = {4:4, 8:2, 32:1} - Batch sizes 1-4: use draft_len=4 - Batch sizes 5-8: use draft_len=2 - Batch sizes 9-32: use draft_len=1 - Batch sizes 33+: use draft_len=0 (implicit, speculation disabled). Mutually exclusive with max_concurrency since draft_len_schedule implicitly support max concurrency control.

field enable_penalty: bool = False

If true, enables the occurrence penalties (repetition / presence / frequency) for one-model speculative decoding. Off by default because the penalties need a [num_seq_slots, vocab_size] occurrence-count workspace that is allocated up front (CUDA graphs capture fixed buffer addresses). While off, a request that asks for any of these penalties is rejected at admission rather than silently decoded without them.

field load_format: str | None = None

The load format of the speculative model.

field markov_head_type: Literal['vanilla', 'gated', 'rnn'] | None = None

Type of the sequential refinement head used within a block. If None, read from the draft model config (dspark_markov_head_type), defaulting to “vanilla”.

field markov_rank: int | None = None

Low-rank dimension of the Markov head logit-bias. If None, read from the draft model config (dspark_markov_rank). 0 disables the head.

field mask_token_id: int | None = None

Token ID used as the mask/noise token for parallel draft prediction. If None, read from the draft model config (dspark_noise_token_id).

field max_concurrency: PositiveInt | None = None

When specified (>0), speculation will be disabled at batch sizes above this value. Otherwise, speculation will always be on. PyTorch backend only. Mutually exclusive with max_concurrency since draft_len_schedule implicitly supports max concurrency control.

field max_draft_len: NonNegativeInt | None = None

The maximum number of draft tokens.

field max_total_draft_tokens: int | None = None

The number of draft tokens in the draft tokens tree. If it’s a linear tree, each draft layer will only generate one draft token. In this case, max_draft_len == max_total_draft_tokens. If it’s a static or dynamic tree, each draft layer may generate more than one draft token. In this case, max_total_draft_tokens >= max_draft_len.

field speculative_model: str | Path | None = None

The speculative (draft) model. Accepts either (1) a HuggingFace Hub model ID (e.g. ‘yuhuili/EAGLE3-LLaMA3.1-Instruct-8B’), which will be automatically downloaded, or (2) a local filesystem path to a downloaded model directory. For one-model MTP, a non-target checkpoint provides either replacement MTP heads or a complete external draft model, depending on the target model implementation. Pointing it at the target checkpoint uses the target’s embedded mtp.* weights.

field target_layer_ids: List[int] | None = None

Target model layer indices whose hidden states are captured for cross-attention in the draft model. If None, read from the draft model config (dspark_target_layer_ids).

field use_rejection_sampling: bool = False

If true, enables rejection sampling for one-model speculative decoding paths when the batch contains any non-greedy request. All-greedy batches always take the argmax fast path regardless of this flag. Set to false (default) to use exact-match verification on non-greedy batches. The non-dynamic-tree one-model path requires FlashInfer.

__init__(**data: Any) → None

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

get_runtime_tokens_per_gen_step(

runtime_draft_len: int,

) → int

Total tokens per gen request for the current runtime draft length.

num_capture_layers() → int

validator set_max_total_draft_tokens  »  all fields

supports_backend(backend: str) → bool

Override if the speculation algorithm does not support a subset of the possible backends.

validator validate_draft_len_schedule_and_sort  »  draft_len_schedule

Validate and sort draft_len_schedule by batch size thresholds.

validator validate_max_concurrency_and_draft_len_schedule_mutually_exclusive  »  all fields

validator validate_rejection_sampling_config  »  all fields

Disable rejection sampling when SA-enhanced configurations are active.

Only silently disable a default-inherited value; an explicit use_rejection_sampling=True is preserved so TorchLlmArgs.validate_speculative_config can raise for the unsupported SA combination.

property draft_is_embedded_in_target: bool

True for the embedded (DeepSeek-V4-Pro) flavour of the DSpark draft.

DSpark ships in two shapes, and they need different runtime plumbing:

• embedded: the draft is the mtp.* namespace of the target checkpoint, built from full target decoder blocks, and served by DSv4DSparkWorker with its own rolling captured-context window.

• standalone: the draft is its own checkpoint with a registry-resolved backbone, served by DFlashWorker and its paged draft KV cache.

Both are decoding_type: DSpark, so every dispatch that must tell them apart – draft-model builder, worker, spec metadata, and the separate-draft-KV-cache decision – reads this one flag instead of re-deriving it. That is what keeps those decisions from drifting apart: a builder and a worker that disagree produce a draft model whose attributes the worker does not have.

The probe is the weight index rather than a config field because the index is authoritative and cannot be left unset; model_type is the fallback for a checkpoint whose index file is absent. Resolution is memoized here and warmed during TorchLlmArgs validation, so the filesystem probe happens once in the main process – not per rank, and never at forward or CUDA-graph-capture time.

property is_linear_tree: bool

property needs_separate_draft_weights: bool

Whether draft weights must be loaded from speculative_model.

This includes external draft models and MTP head replacement checkpoints.

property spec_dec_mode

property tokens_per_gen_step: int

DSpark needs K+1 tokens per gen request (K drafts + 1 bonus).

The draft produces its own mask queries internally; passing mask fillers through the target is pure wasted work at large batch size.

property uses_external_draft_model: bool

Whether speculative_model contains an external draft model.

property uses_replacement_heads: bool

Whether speculative_model contains replacement MTP heads.