DFlashDecodingConfig#

class tensorrt_llm.llmapi.DFlashDecodingConfig( *, max_draft_len: Annotated[int, Ge(ge=0)] | None = None, max_total_draft_tokens: int | None = None, speculative_model: str | Path | None = None, max_concurrency: Annotated[int, Gt(gt=0)] | None = None, draft_len_schedule: dict[int, int] | None = None, load_format: str | None = None, acceptance_rate_window_size: Annotated[int, Ge(ge=0)] | None = None, acceptance_rate_threshold: Annotated[float | None, Ge(ge=0.0), Le(le=1.0)] = None, use_rejection_sampling: bool = False, allow_advanced_sampling: bool = False, advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL, enable_penalty: bool = False, mask_token_id: int | None = None, target_layer_ids: List[int] | None = None, decoding_type: Literal['DFlash'] = 'DFlash', attention_backend: Literal['VANILLA', 'TRTLLM'] = 'VANILLA', )[source]#

Bases: DecodingBaseConfig

Configuration for DFlash speculative decoding.

DFlash is a target-dependent speculative decoding method that uses hidden states from specific target model layers as cross-attention context in the draft model to predict multiple draft tokens in parallel.

Key features: - Target-dependent: uses hidden states from target model layers - Parallel prediction: all K draft tokens in one forward pass - Cross-attention: draft model attends to target hidden states

Reference: https://arxiv.org/pdf/2602.06036

field acceptance_rate_threshold: float | None = None#

The threshold for average true acceptance rate (accepted_draft_tokens / drafted_tokens); speculation will be disabled permanently once the rolling average over the last N speculation-enabled decoding iterations (N = acceptance_rate_window_size) drops below this value.

Constraints:

ge = 0.0
le = 1.0

field acceptance_rate_window_size: NonNegativeInt | None = None#: The rolling average window size (N) for acceptance rate across speculation-enabled decoding iterations. If not set or set to 0, the feature is disabled. PyTorch backend only.

field advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL#: Deploy-time specialization of the one-model advanced sampler that skips disabled filter kernels. FULL (default): per-row top_k/top_p. NO_TOPK: skip top_k. NO_TOPP: skip top_p. NO_TOPK_NO_TOPP: skip both.

field allow_advanced_sampling: bool = False#: DEPRECATED: no-op kept for backward compatibility. Will be removed in a future release. Non-greedy sampling is now auto-detected per request; this flag no longer has any effect.

field attention_backend: Literal['VANILLA', 'TRTLLM'] = 'VANILLA'#: Attention backend for DFlash pooled-context cross-attention. This is independent of the backend used to construct the drafter’s standard attention modules. TRTLLM requires FlashInfer and an NVIDIA Blackwell GPU with SM100 or SM103, and uses generated FMHA kernels with a private paged context cache; VANILLA uses FlashAttention with a contiguous cache.

field decoding_type: Literal['DFlash'] = 'DFlash'#

field draft_len_schedule: dict[int, int] | None = None#: Developer interface: dynamically adjust draft length based on active batch size in runtime.Maps batch size to draft lengths.For example: draft_len_schedule = {4:4, 8:2, 32:1} - Batch sizes 1-4: use draft_len=4 - Batch sizes 5-8: use draft_len=2 - Batch sizes 9-32: use draft_len=1 - Batch sizes 33+: use draft_len=0 (implicit, speculation disabled). Mutually exclusive with max_concurrency since draft_len_schedule implicitly support max concurrency control.

field enable_penalty: bool = False#: If true, enables the occurrence penalties (repetition / presence / frequency) for one-model speculative decoding. Off by default because the penalties need a [num_seq_slots, vocab_size] occurrence-count workspace that is allocated up front (CUDA graphs capture fixed buffer addresses). While off, a request that asks for any of these penalties is rejected at admission rather than silently decoded without them.

field load_format: str | None = None#: The load format of the speculative model.

field mask_token_id: int | None = None#: The token ID used as a mask token for parallel draft prediction. If None, it will be read from the draft model config (dflash_config.mask_token_id).

field max_concurrency: PositiveInt | None = None#: When specified (>0), speculation will be disabled at batch sizes above this value. Otherwise, speculation will always be on. PyTorch backend only. Mutually exclusive with max_concurrency since draft_len_schedule implicitly supports max concurrency control.

field max_draft_len: NonNegativeInt | None = None#: The maximum number of draft tokens.

field max_total_draft_tokens: int | None = None#: The number of draft tokens in the draft tokens tree. If it’s a linear tree, each draft layer will only generate one draft token. In this case, max_draft_len == max_total_draft_tokens. If it’s a static or dynamic tree, each draft layer may generate more than one draft token. In this case, max_total_draft_tokens >= max_draft_len.

field speculative_model: str | Path | None = None#: The speculative (draft) model. Accepts either (1) a HuggingFace Hub model ID (e.g. ‘yuhuili/EAGLE3-LLaMA3.1-Instruct-8B’), which will be automatically downloaded, or (2) a local filesystem path to a downloaded model directory. For one-model MTP, a non-target checkpoint provides either replacement MTP heads or a complete external draft model, depending on the target model implementation. Pointing it at the target checkpoint uses the target’s embedded mtp.* weights.

field target_layer_ids: List[int] | None = None#: List of target model layer indices whose hidden states are captured for cross-attention in the draft model. If None, read from the draft model config (dflash_config.target_layer_ids).

field use_rejection_sampling: bool = False#: If true, enables rejection sampling for one-model speculative decoding paths when the batch contains any non-greedy request. All-greedy batches always take the argmax fast path regardless of this flag. Set to false (default) to use exact-match verification on non-greedy batches. The non-dynamic-tree one-model path requires FlashInfer.

class Config#

Bases: object

extra = 'forbid'#

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

classmethod construct(

_fields_set: set[str] | None = None,

**values: Any,

) → Self#

Returns a copy of the model.

!!! warning “Deprecated”: This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

include – Optional set or mapping specifying which fields to include in the copied model.
exclude – Optional set or mapping specifying which fields to exclude in the copied model.
update – Optional dictionary of field-value pairs to override field values in the copied model.
deep – If True, the values of fields that are Pydantic models will be deep-copied.

Returns:

A copy of the model with included, excluded and updated fields as specified.

dict( *, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, by_alias: bool = False, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, ) → Dict[str, Any]#

classmethod from_orm(obj: Any) → Self#

get_runtime_tokens_per_gen_step( runtime_draft_len: int, ) → int[source]#: DFlash needs K+1 runtime tokens per gen request (K drafts + 1 bonus).

json(

*,

include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None,

exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None,

by_alias: bool = False,

exclude_unset: bool = False,

exclude_defaults: bool = False,

exclude_none: bool = False,

encoder: Callable[[Any], Any] | None = PydanticUndefined,

models_as_dict: bool = PydanticUndefined,

**dumps_kwargs: Any,

) → str#

classmethod model_construct(

_fields_set: set[str] | None = None,

**values: Any,

) → Self#

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note: model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.

Parameters:

_fields_set – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.
values – Trusted or pre-validated data dictionary.

Returns:

A new instance of the Model class with validated data.

model_copy( *, update: Mapping[str, Any] | None = None, deep: bool = False, ) → Self#

!!! abstract “Usage Documentation”: [model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note: The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

update – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.
deep – Set to True to make a deep copy of the model.

Returns:

New model instance.

!!! abstract “Usage Documentation”: [model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

mode – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.
include – A set of fields to include in the output.
exclude – A set of fields to exclude from the output.
context – Additional context to pass to the serializer.
by_alias – Whether to use the field’s alias in the dictionary key if defined.
exclude_unset – Whether to exclude fields that have not been explicitly set.
exclude_defaults – Whether to exclude fields that are set to their default value.
exclude_none – Whether to exclude fields that have a value of None.
exclude_computed_fields – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization – Whether to use model and dataclass polymorphic serialization for this call.

Returns:

A dictionary representation of the model.

!!! abstract “Usage Documentation”: [model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

indent – Indentation to use in the JSON output. If None is passed, the output will be compact.
ensure_ascii – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.
include – Field(s) to include in the JSON output.
exclude – Field(s) to exclude from the JSON output.
context – Additional context to pass to the serializer.
by_alias – Whether to serialize using field aliases.
exclude_unset – Whether to exclude fields that have not been explicitly set.
exclude_defaults – Whether to exclude fields that are set to their default value.
exclude_none – Whether to exclude fields that have a value of None.
exclude_computed_fields – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization – Whether to use model and dataclass polymorphic serialization for this call.

Returns:

A JSON string representation of the model.

classmethod model_json_schema( by_alias: bool = True, ref_template: str = '#/$defs/{model}', schema_generator: type[~pydantic.json_schema.GenerateJsonSchema] = <class 'pydantic.json_schema.GenerateJsonSchema'>, mode: ~typing.Literal['validation', 'serialization'] = 'validation', *, union_format: ~typing.Literal['any_of', 'primitive_type_array'] = 'any_of', ) → dict[str, Any]#

Generates a JSON schema for a model class.

Parameters:

by_alias – Whether to use attribute aliases or not.
ref_template – The reference template.
union_format –
The format to use when combining schemas from unions together. Can be one of:
- ’any_of’: Use the [anyOf](https://json-schema.org/understanding-json-schema/reference/combining#anyOf)
keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.
schema_generator – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications
mode – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

classmethod model_parametrized_name( params: tuple[type[Any], ...], ) → str#

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:: params – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.
Returns:: String representing the new class where params are passed to cls as type variables.
Raises:: TypeError – Raised when trying to generate concrete names for non-generic models.

model_post_init(context: Any, /) → None#

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

self – The BaseModel instance.
context – The context.

classmethod model_rebuild( *, force: bool = False, raise_errors: bool = True, _parent_namespace_depth: int = 2, _types_namespace: MappingNamespace | None = None, ) → bool | None#

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

force – Whether to force the rebuilding of the model schema, defaults to False.
raise_errors – Whether to raise errors, defaults to True.
_parent_namespace_depth – The depth level of the parent namespace, defaults to 2.
_types_namespace – The types namespace, defaults to None.

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

Validate a pydantic model instance.

Parameters:

obj – The object to validate.
strict – Whether to enforce types strictly.
extra – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
from_attributes – Whether to extract data from object attributes.
context – Additional context to pass to the validator.
by_alias – Whether to use the field’s alias when validating against the provided input data.
by_name – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

!!! abstract “Usage Documentation”: [JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

json_data – The JSON data to validate.
strict – Whether to enforce types strictly.
extra – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context – Extra variables to pass to the validator.
by_alias – Whether to use the field’s alias when validating against the provided input data.
by_name – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

Validate the given object with string data against the Pydantic model.

Parameters:

obj – The object containing string data to validate.
strict – Whether to enforce types strictly.
extra – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context – Extra variables to pass to the validator.
by_alias – Whether to use the field’s alias when validating against the provided input data.
by_name – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

num_capture_layers() → int#

classmethod parse_file( path: str | Path, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False, ) → Self#

classmethod parse_obj(obj: Any) → Self#

classmethod parse_raw( b: str | bytes, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False, ) → Self#

classmethod schema( by_alias: bool = True, ref_template: str = '#/$defs/{model}', ) → Dict[str, Any]#

classmethod schema_json(

*,

by_alias: bool = True,

ref_template: str = '#/$defs/{model}',

**dumps_kwargs: Any,

) → str#

validator set_max_total_draft_tokens » all fields[source]#

supports_backend(backend: str) → bool[source]#: Override if the speculation algorithm does not support a subset of the possible backends.

classmethod update_forward_refs(

**localns: Any,

) → None#

classmethod validate(value: Any) → Self#

validator validate_draft_len_schedule_and_sort » draft_len_schedule#: Validate and sort draft_len_schedule by batch size thresholds.

validator validate_max_concurrency_and_draft_len_schedule_mutually_exclusive » all fields#

validator validate_rejection_sampling_config » all fields#

Disable rejection sampling when SA-enhanced configurations are active.

Only silently disable a default-inherited value; an explicit use_rejection_sampling=True is preserved so TorchLlmArgs.validate_speculative_config can raise for the unsupported SA combination.

property is_linear_tree: bool#

model_computed_fields = {}#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property model_extra: dict[str, Any] | None#

Get extra fields set during validation.

Returns:: A dictionary of extra fields, or None if config.extra is not set to “allow”.

model_fields = {'acceptance_rate_threshold': FieldInfo(annotation=Union[float, NoneType], required=False, default=None, description='The threshold for average true acceptance rate (accepted_draft_tokens / drafted_tokens); speculation will be disabled permanently once the rolling average over the last N speculation-enabled decoding iterations (N = acceptance_rate_window_size) drops below this value. ', metadata=[Ge(ge=0.0), Le(le=1.0)]), 'acceptance_rate_window_size': FieldInfo(annotation=Union[Annotated[int, Ge], NoneType], required=False, default=None, description='The rolling average window size (N) for acceptance rate across speculation-enabled decoding iterations. If not set or set to 0, the feature is disabled. PyTorch backend only.'), 'advanced_sampling_mode': FieldInfo(annotation=AdvancedSamplingMode, required=False, default=<AdvancedSamplingMode.FULL: 'full'>, description='Deploy-time specialization of the one-model advanced sampler that skips disabled filter kernels. FULL (default): per-row top_k/top_p. NO_TOPK: skip top_k. NO_TOPP: skip top_p. NO_TOPK_NO_TOPP: skip both.'), 'allow_advanced_sampling': FieldInfo(annotation=bool, required=False, default=False, description='DEPRECATED: no-op kept for backward compatibility. Will be removed in a future release. Non-greedy sampling is now auto-detected per request; this flag no longer has any effect.', json_schema_extra={'status': 'deprecated'}), 'attention_backend': FieldInfo(annotation=Literal['VANILLA', 'TRTLLM'], required=False, default='VANILLA', description="Attention backend for DFlash pooled-context cross-attention. This is independent of the backend used to construct the drafter's standard attention modules. TRTLLM requires FlashInfer and an NVIDIA Blackwell GPU with SM100 or SM103, and uses generated FMHA kernels with a private paged context cache; VANILLA uses FlashAttention with a contiguous cache."), 'decoding_type': FieldInfo(annotation=Literal['DFlash'], required=False, default='DFlash'), 'draft_len_schedule': FieldInfo(annotation=Union[dict[int, int], NoneType], required=False, default=None, description='Developer interface: dynamically adjust draft length based on active batch size in runtime.Maps batch size to draft lengths.For example: draft_len_schedule = {4:4, 8:2, 32:1} - Batch sizes 1-4: use draft_len=4 - Batch sizes 5-8: use draft_len=2 - Batch sizes 9-32: use draft_len=1 - Batch sizes 33+: use draft_len=0 (implicit, speculation disabled). Mutually exclusive with max_concurrency since draft_len_schedule implicitly support max concurrency control.'), 'enable_penalty': FieldInfo(annotation=bool, required=False, default=False, description='If true, enables the occurrence penalties (repetition / presence / frequency) for one-model speculative decoding. Off by default because the penalties need a [num_seq_slots, vocab_size] occurrence-count workspace that is allocated up front (CUDA graphs capture fixed buffer addresses). While off, a request that asks for any of these penalties is rejected at admission rather than silently decoded without them.', json_schema_extra={'status': 'prototype'}), 'load_format': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, description='The load format of the speculative model.'), 'mask_token_id': FieldInfo(annotation=Union[int, NoneType], required=False, default=None, description='The token ID used as a mask token for parallel draft prediction. If None, it will be read from the draft model config (dflash_config.mask_token_id).'), 'max_concurrency': FieldInfo(annotation=Union[Annotated[int, Gt], NoneType], required=False, default=None, description='When specified (>0), speculation will be disabled at batch sizes above this value. Otherwise, speculation will always be on. PyTorch backend only. Mutually exclusive with max_concurrency since draft_len_schedule implicitly supports max concurrency control.'), 'max_draft_len': FieldInfo(annotation=Union[Annotated[int, Ge], NoneType], required=False, default=None, description='The maximum number of draft tokens.'), 'max_total_draft_tokens': FieldInfo(annotation=Union[int, NoneType], required=False, default=None, description="The number of draft tokens in the draft tokens tree. If it's a linear tree, each draft layer will only generate one draft token. In this case, max_draft_len == max_total_draft_tokens. If it's a static or dynamic tree, each draft layer may generate more than one draft token. In this case, max_total_draft_tokens >= max_draft_len."), 'speculative_model': FieldInfo(annotation=Union[str, Path, NoneType], required=False, default=None, alias_priority=2, validation_alias=AliasChoices(choices=['speculative_model', 'speculative_model_dir']), description="The speculative (draft) model. Accepts either (1) a HuggingFace Hub model ID (e.g. 'yuhuili/EAGLE3-LLaMA3.1-Instruct-8B'), which will be automatically downloaded, or (2) a local filesystem path to a downloaded model directory. For one-model MTP, a non-target checkpoint provides either replacement MTP heads or a complete external draft model, depending on the target model implementation. Pointing it at the target checkpoint uses the target's embedded mtp.* weights."), 'target_layer_ids': FieldInfo(annotation=Union[List[int], NoneType], required=False, default=None, description='List of target model layer indices whose hidden states are captured for cross-attention in the draft model. If None, read from the draft model config (dflash_config.target_layer_ids).'), 'use_rejection_sampling': FieldInfo(annotation=bool, required=False, default=False, description='If true, enables rejection sampling for one-model speculative decoding paths when the batch contains any non-greedy request. All-greedy batches always take the argmax fast path regardless of this flag. Set to false (default) to use exact-match verification on non-greedy batches. The non-dynamic-tree one-model path requires FlashInfer.', json_schema_extra={'status': 'prototype'})}#

property model_fields_set: set[str]#

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,: i.e. that were not filled from defaults.

property needs_separate_draft_weights: bool#

Whether draft weights must be loaded from speculative_model.

This includes external draft models and MTP head replacement checkpoints.

property spec_dec_mode#

property tokens_per_gen_step: int#

DFlash only needs K+1 tokens per gen request (K drafts + 1 bonus).

The draft produces its own mask queries internally; passing mask fillers through the target is pure wasted work at large batch size.

property uses_external_draft_model: bool#: Whether speculative_model contains an external draft model.

property uses_replacement_heads: bool#: Whether speculative_model contains replacement MTP heads.