EagleDecodingConfig#

class tensorrt_llm.llmapi.EagleDecodingConfig( *, max_draft_len: Annotated[int, Ge(ge=0)] | None = None, max_total_draft_tokens: int | None = None, speculative_model: str | Path | None = None, max_concurrency: Annotated[int, Gt(gt=0)] | None = None, draft_len_schedule: dict[int, int] | None = None, load_format: str | None = None, acceptance_rate_window_size: Annotated[int, Ge(ge=0)] | None = None, acceptance_rate_threshold: Annotated[float | None, Ge(ge=0.0), Le(le=1.0)] = None, use_rejection_sampling: bool = False, allow_advanced_sampling: bool = False, advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL, enable_penalty: bool = False, decoding_type: Literal['Eagle'] = 'Eagle', eagle_choices: List[List[int]] | None = None, greedy_sampling: bool | None = True, posterior_threshold: float | None = None, use_dynamic_tree: bool | None = False, dynamic_tree_max_topK: int | None = None, num_eagle_layers: int | None = None, max_non_leaves_per_layer: int | None = None, eagle3_one_model: bool | None = True, eagle3_layers_to_capture: Set[int] | None = None, eagle3_model_arch: Literal['llama3', 'mistral_large3'] = 'llama3', )[source]#

Bases: DecodingBaseConfig

field acceptance_rate_threshold: float | None = None#

The threshold for average true acceptance rate (accepted_draft_tokens / drafted_tokens); speculation will be disabled permanently once the rolling average over the last N speculation-enabled decoding iterations (N = acceptance_rate_window_size) drops below this value.

Constraints:

ge = 0.0
le = 1.0

field acceptance_rate_window_size: NonNegativeInt | None = None#: The rolling average window size (N) for acceptance rate across speculation-enabled decoding iterations. If not set or set to 0, the feature is disabled. PyTorch backend only.

field advanced_sampling_mode: AdvancedSamplingMode = AdvancedSamplingMode.FULL#: Deploy-time specialization of the one-model advanced sampler that skips disabled filter kernels. FULL (default): per-row top_k/top_p. NO_TOPK: skip top_k. NO_TOPP: skip top_p. NO_TOPK_NO_TOPP: skip both.

field allow_advanced_sampling: bool = False#: DEPRECATED: no-op kept for backward compatibility. Will be removed in a future release. Non-greedy sampling is now auto-detected per request; this flag no longer has any effect.

field decoding_type: Literal['Eagle'] = 'Eagle'#

field draft_len_schedule: dict[int, int] | None = None#: Developer interface: dynamically adjust draft length based on active batch size in runtime.Maps batch size to draft lengths.For example: draft_len_schedule = {4:4, 8:2, 32:1} - Batch sizes 1-4: use draft_len=4 - Batch sizes 5-8: use draft_len=2 - Batch sizes 9-32: use draft_len=1 - Batch sizes 33+: use draft_len=0 (implicit, speculation disabled). Mutually exclusive with max_concurrency since draft_len_schedule implicitly support max concurrency control.

field dynamic_tree_max_topK: int | None = None#: The topK value for each layer when dynamic tree is enabled. Required when use_dynamic_tree is True; ignored (with a warning) when use_dynamic_tree is False.

field eagle3_layers_to_capture: Set[int] | None = None#: Target model layer indices to capture hidden states from for the EAGLE3 draft model. Defaults to {1, num_layers//2-1, num_layers-4}.

field eagle3_model_arch: Literal['llama3', 'mistral_large3'] = 'llama3'#: The model architecture of the eagle3 model.

field eagle3_one_model: bool | None = True#: Always uses the one-model implementation (draft as submodule). Setting False is ignored and falls back to True; the two-model path is deprecated and will be removed in a future release.

field eagle_choices: List[List[int]] | None = None#: Static tree structure for draft token generation. Each sublist represents a path in the tree. Mutually exclusive with use_dynamic_tree.

field enable_penalty: bool = False#: If true, enables the occurrence penalties (repetition / presence / frequency) for one-model speculative decoding. Off by default because the penalties need a [num_seq_slots, vocab_size] occurrence-count workspace that is allocated up front (CUDA graphs capture fixed buffer addresses). While off, a request that asks for any of these penalties is rejected at admission rather than silently decoded without them.

field greedy_sampling: bool | None = True#: Whether to use greedy sampling (Top-1 with token equality acceptance) or typical acceptance with multinomial sampling.

field load_format: str | None = None#: The load format of the speculative model.

field max_concurrency: PositiveInt | None = None#: When specified (>0), speculation will be disabled at batch sizes above this value. Otherwise, speculation will always be on. PyTorch backend only. Mutually exclusive with max_concurrency since draft_len_schedule implicitly supports max concurrency control.

field max_draft_len: NonNegativeInt | None = None#: The maximum number of draft tokens.

field max_non_leaves_per_layer: int | None = None#: The number of non-leaves in each layer.

field max_total_draft_tokens: int | None = None#: The number of draft tokens in the draft tokens tree. If it’s a linear tree, each draft layer will only generate one draft token. In this case, max_draft_len == max_total_draft_tokens. If it’s a static or dynamic tree, each draft layer may generate more than one draft token. In this case, max_total_draft_tokens >= max_draft_len.

field num_eagle_layers: int | None = None#: Deprecated TensorRT-only field with different semantics from draft model layer count. Do not use on the PyTorch backend.

field posterior_threshold: float | None = None#: Minimum token probability threshold for typical acceptance. Corresponds to epsilon in https://arxiv.org/pdf/2401.10774.

field speculative_model: str | Path | None = None#: The speculative (draft) model. Accepts either (1) a HuggingFace Hub model ID (e.g. ‘yuhuili/EAGLE3-LLaMA3.1-Instruct-8B’), which will be automatically downloaded, or (2) a local filesystem path to a downloaded model directory. For one-model MTP, a non-target checkpoint provides either replacement MTP heads or a complete external draft model, depending on the target model implementation. Pointing it at the target checkpoint uses the target’s embedded mtp.* weights.

field use_dynamic_tree: bool | None = False#: Whether to use dynamic tree (Eagle-2 algorithm). Mutually exclusive with eagle_choices.

field use_rejection_sampling: bool = False#: If true, enables rejection sampling for one-model speculative decoding paths when the batch contains any non-greedy request. All-greedy batches always take the argmax fast path regardless of this flag. Set to false (default) to use exact-match verification on non-greedy batches. The non-dynamic-tree one-model path requires FlashInfer.

class Config#

Bases: object

extra = 'forbid'#

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

check_eagle_choices()[source]#

classmethod construct(

_fields_set: set[str] | None = None,

**values: Any,

) → Self#

Returns a copy of the model.

!!! warning “Deprecated”: This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

include – Optional set or mapping specifying which fields to include in the copied model.
exclude – Optional set or mapping specifying which fields to exclude in the copied model.
update – Optional dictionary of field-value pairs to override field values in the copied model.
deep – If True, the values of fields that are Pydantic models will be deep-copied.

Returns:

A copy of the model with included, excluded and updated fields as specified.

dict( *, include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None, by_alias: bool = False, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, ) → Dict[str, Any]#

classmethod from_orm(obj: Any) → Self#

get_runtime_tokens_per_gen_step( runtime_draft_len: int, ) → int#: Total tokens per gen request for the current runtime draft length.

json(

*,

include: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None,

exclude: set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None = None,

by_alias: bool = False,

exclude_unset: bool = False,

exclude_defaults: bool = False,

exclude_none: bool = False,

encoder: Callable[[Any], Any] | None = PydanticUndefined,

models_as_dict: bool = PydanticUndefined,

**dumps_kwargs: Any,

) → str#

classmethod model_construct(

_fields_set: set[str] | None = None,

**values: Any,

) → Self#

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note: model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.

Parameters:

_fields_set – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.
values – Trusted or pre-validated data dictionary.

Returns:

A new instance of the Model class with validated data.

model_copy( *, update: Mapping[str, Any] | None = None, deep: bool = False, ) → Self#

!!! abstract “Usage Documentation”: [model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note: The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

update – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.
deep – Set to True to make a deep copy of the model.

Returns:

New model instance.

!!! abstract “Usage Documentation”: [model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

mode – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.
include – A set of fields to include in the output.
exclude – A set of fields to exclude from the output.
context – Additional context to pass to the serializer.
by_alias – Whether to use the field’s alias in the dictionary key if defined.
exclude_unset – Whether to exclude fields that have not been explicitly set.
exclude_defaults – Whether to exclude fields that are set to their default value.
exclude_none – Whether to exclude fields that have a value of None.
exclude_computed_fields – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization – Whether to use model and dataclass polymorphic serialization for this call.

Returns:

A dictionary representation of the model.

!!! abstract “Usage Documentation”: [model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

indent – Indentation to use in the JSON output. If None is passed, the output will be compact.
ensure_ascii – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.
include – Field(s) to include in the JSON output.
exclude – Field(s) to exclude from the JSON output.
context – Additional context to pass to the serializer.
by_alias – Whether to serialize using field aliases.
exclude_unset – Whether to exclude fields that have not been explicitly set.
exclude_defaults – Whether to exclude fields that are set to their default value.
exclude_none – Whether to exclude fields that have a value of None.
exclude_computed_fields – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.
round_trip – If True, dumped values should be valid as input for non-idempotent types such as Json[T].
warnings – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].
fallback – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.
serialize_as_any – Whether to serialize fields with duck-typing serialization behavior.
polymorphic_serialization – Whether to use model and dataclass polymorphic serialization for this call.

Returns:

A JSON string representation of the model.

classmethod model_json_schema( by_alias: bool = True, ref_template: str = '#/$defs/{model}', schema_generator: type[~pydantic.json_schema.GenerateJsonSchema] = <class 'pydantic.json_schema.GenerateJsonSchema'>, mode: ~typing.Literal['validation', 'serialization'] = 'validation', *, union_format: ~typing.Literal['any_of', 'primitive_type_array'] = 'any_of', ) → dict[str, Any]#

Generates a JSON schema for a model class.

Parameters:

by_alias – Whether to use attribute aliases or not.
ref_template – The reference template.
union_format –
The format to use when combining schemas from unions together. Can be one of:
- ’any_of’: Use the [anyOf](https://json-schema.org/understanding-json-schema/reference/combining#anyOf)
keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.
schema_generator – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications
mode – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

classmethod model_parametrized_name( params: tuple[type[Any], ...], ) → str#

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:: params – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.
Returns:: String representing the new class where params are passed to cls as type variables.
Raises:: TypeError – Raised when trying to generate concrete names for non-generic models.

model_post_init(context: Any, /) → None#

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

self – The BaseModel instance.
context – The context.

classmethod model_rebuild( *, force: bool = False, raise_errors: bool = True, _parent_namespace_depth: int = 2, _types_namespace: MappingNamespace | None = None, ) → bool | None#

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

force – Whether to force the rebuilding of the model schema, defaults to False.
raise_errors – Whether to raise errors, defaults to True.
_parent_namespace_depth – The depth level of the parent namespace, defaults to 2.
_types_namespace – The types namespace, defaults to None.

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

Validate a pydantic model instance.

Parameters:

obj – The object to validate.
strict – Whether to enforce types strictly.
extra – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
from_attributes – Whether to extract data from object attributes.
context – Additional context to pass to the validator.
by_alias – Whether to use the field’s alias when validating against the provided input data.
by_name – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

!!! abstract “Usage Documentation”: [JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

json_data – The JSON data to validate.
strict – Whether to enforce types strictly.
extra – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context – Extra variables to pass to the validator.
by_alias – Whether to use the field’s alias when validating against the provided input data.
by_name – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

Validate the given object with string data against the Pydantic model.

Parameters:

obj – The object containing string data to validate.
strict – Whether to enforce types strictly.
extra – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.
context – Extra variables to pass to the validator.
by_alias – Whether to use the field’s alias when validating against the provided input data.
by_name – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

classmethod parse_file( path: str | Path, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False, ) → Self#

classmethod parse_obj(obj: Any) → Self#

classmethod parse_raw( b: str | bytes, *, content_type: str | None = None, encoding: str = 'utf8', proto: DeprecatedParseProtocol | None = None, allow_pickle: bool = False, ) → Self#

classmethod schema( by_alias: bool = True, ref_template: str = '#/$defs/{model}', ) → Dict[str, Any]#

classmethod schema_json(

*,

by_alias: bool = True,

ref_template: str = '#/$defs/{model}',

**dumps_kwargs: Any,

) → str#

supports_backend(backend: str) → bool#: Override if the speculation algorithm does not support a subset of the possible backends.

classmethod update_forward_refs(

**localns: Any,

) → None#

classmethod validate(value: Any) → Self#

validator validate_draft_len_schedule_and_sort » draft_len_schedule#: Validate and sort draft_len_schedule by batch size thresholds.

validator validate_eagle_choices » eagle_choices[source]#

validator validate_eagle_config » all fields[source]#

validator validate_max_concurrency_and_draft_len_schedule_mutually_exclusive » all fields#

validator validate_rejection_sampling_config » all fields#

Disable rejection sampling when SA-enhanced configurations are active.

Only silently disable a default-inherited value; an explicit use_rejection_sampling=True is preserved so TorchLlmArgs.validate_speculative_config can raise for the unsupported SA combination.

validator validate_speculative_model » all fields[source]#

property is_linear_tree: bool#

model_computed_fields = {}#

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property model_extra: dict[str, Any] | None#

Get extra fields set during validation.

Returns:: A dictionary of extra fields, or None if config.extra is not set to “allow”.

model_fields = {'acceptance_rate_threshold': FieldInfo(annotation=Union[float, NoneType], required=False, default=None, description='The threshold for average true acceptance rate (accepted_draft_tokens / drafted_tokens); speculation will be disabled permanently once the rolling average over the last N speculation-enabled decoding iterations (N = acceptance_rate_window_size) drops below this value. ', metadata=[Ge(ge=0.0), Le(le=1.0)]), 'acceptance_rate_window_size': FieldInfo(annotation=Union[Annotated[int, Ge], NoneType], required=False, default=None, description='The rolling average window size (N) for acceptance rate across speculation-enabled decoding iterations. If not set or set to 0, the feature is disabled. PyTorch backend only.'), 'advanced_sampling_mode': FieldInfo(annotation=AdvancedSamplingMode, required=False, default=<AdvancedSamplingMode.FULL: 'full'>, description='Deploy-time specialization of the one-model advanced sampler that skips disabled filter kernels. FULL (default): per-row top_k/top_p. NO_TOPK: skip top_k. NO_TOPP: skip top_p. NO_TOPK_NO_TOPP: skip both.'), 'allow_advanced_sampling': FieldInfo(annotation=bool, required=False, default=False, description='DEPRECATED: no-op kept for backward compatibility. Will be removed in a future release. Non-greedy sampling is now auto-detected per request; this flag no longer has any effect.', json_schema_extra={'status': 'deprecated'}), 'decoding_type': FieldInfo(annotation=Literal['Eagle'], required=False, default='Eagle'), 'draft_len_schedule': FieldInfo(annotation=Union[dict[int, int], NoneType], required=False, default=None, description='Developer interface: dynamically adjust draft length based on active batch size in runtime.Maps batch size to draft lengths.For example: draft_len_schedule = {4:4, 8:2, 32:1} - Batch sizes 1-4: use draft_len=4 - Batch sizes 5-8: use draft_len=2 - Batch sizes 9-32: use draft_len=1 - Batch sizes 33+: use draft_len=0 (implicit, speculation disabled). Mutually exclusive with max_concurrency since draft_len_schedule implicitly support max concurrency control.'), 'dynamic_tree_max_topK': FieldInfo(annotation=Union[int, NoneType], required=False, default=None, description='The topK value for each layer when dynamic tree is enabled. Required when use_dynamic_tree is True; ignored (with a warning) when use_dynamic_tree is False.'), 'eagle3_layers_to_capture': FieldInfo(annotation=Union[Set[int], NoneType], required=False, default=None, description='Target model layer indices to capture hidden states from for the EAGLE3 draft model. Defaults to {1, num_layers//2-1, num_layers-4}.'), 'eagle3_model_arch': FieldInfo(annotation=Literal['llama3', 'mistral_large3'], required=False, default='llama3', description='The model architecture of the eagle3 model.'), 'eagle3_one_model': FieldInfo(annotation=Union[bool, NoneType], required=False, default=True, description='Always uses the one-model implementation (draft as submodule). Setting False is ignored and falls back to True; the two-model path is deprecated and will be removed in a future release.'), 'eagle_choices': FieldInfo(annotation=Union[List[List[int]], NoneType], required=False, default=None, description='Static tree structure for draft token generation. Each sublist represents a path in the tree. Mutually exclusive with use_dynamic_tree.'), 'enable_penalty': FieldInfo(annotation=bool, required=False, default=False, description='If true, enables the occurrence penalties (repetition / presence / frequency) for one-model speculative decoding. Off by default because the penalties need a [num_seq_slots, vocab_size] occurrence-count workspace that is allocated up front (CUDA graphs capture fixed buffer addresses). While off, a request that asks for any of these penalties is rejected at admission rather than silently decoded without them.', json_schema_extra={'status': 'prototype'}), 'greedy_sampling': FieldInfo(annotation=Union[bool, NoneType], required=False, default=True, description='Whether to use greedy sampling (Top-1 with token equality acceptance) or typical acceptance with multinomial sampling.'), 'load_format': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, description='The load format of the speculative model.'), 'max_concurrency': FieldInfo(annotation=Union[Annotated[int, Gt], NoneType], required=False, default=None, description='When specified (>0), speculation will be disabled at batch sizes above this value. Otherwise, speculation will always be on. PyTorch backend only. Mutually exclusive with max_concurrency since draft_len_schedule implicitly supports max concurrency control.'), 'max_draft_len': FieldInfo(annotation=Union[Annotated[int, Ge], NoneType], required=False, default=None, description='The maximum number of draft tokens.'), 'max_non_leaves_per_layer': FieldInfo(annotation=Union[int, NoneType], required=False, default=None, description='The number of non-leaves in each layer.'), 'max_total_draft_tokens': FieldInfo(annotation=Union[int, NoneType], required=False, default=None, description="The number of draft tokens in the draft tokens tree. If it's a linear tree, each draft layer will only generate one draft token. In this case, max_draft_len == max_total_draft_tokens. If it's a static or dynamic tree, each draft layer may generate more than one draft token. In this case, max_total_draft_tokens >= max_draft_len."), 'num_eagle_layers': FieldInfo(annotation=Union[int, NoneType], required=False, default=None, description='Deprecated TensorRT-only field with different semantics from draft model layer count. Do not use on the PyTorch backend.'), 'posterior_threshold': FieldInfo(annotation=Union[float, NoneType], required=False, default=None, description='Minimum token probability threshold for typical acceptance. Corresponds to epsilon in https://arxiv.org/pdf/2401.10774.'), 'speculative_model': FieldInfo(annotation=Union[str, Path, NoneType], required=False, default=None, alias_priority=2, validation_alias=AliasChoices(choices=['speculative_model', 'speculative_model_dir']), description="The speculative (draft) model. Accepts either (1) a HuggingFace Hub model ID (e.g. 'yuhuili/EAGLE3-LLaMA3.1-Instruct-8B'), which will be automatically downloaded, or (2) a local filesystem path to a downloaded model directory. For one-model MTP, a non-target checkpoint provides either replacement MTP heads or a complete external draft model, depending on the target model implementation. Pointing it at the target checkpoint uses the target's embedded mtp.* weights."), 'use_dynamic_tree': FieldInfo(annotation=Union[bool, NoneType], required=False, default=False, description='Whether to use dynamic tree (Eagle-2 algorithm). Mutually exclusive with eagle_choices.'), 'use_rejection_sampling': FieldInfo(annotation=bool, required=False, default=False, description='If true, enables rejection sampling for one-model speculative decoding paths when the batch contains any non-greedy request. All-greedy batches always take the argmax fast path regardless of this flag. Set to false (default) to use exact-match verification on non-greedy batches. The non-dynamic-tree one-model path requires FlashInfer.', json_schema_extra={'status': 'prototype'})}#

property model_fields_set: set[str]#

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,: i.e. that were not filled from defaults.

property needs_separate_draft_weights: bool#

Whether draft weights must be loaded from speculative_model.

This includes external draft models and MTP head replacement checkpoints.

property num_capture_layers: int#: Returns the number of layers to capture of the target model. If eagle3_layers_to_capture is not None, return the length of the set. Otherwise, assume Eagle3 base set and return 3.

property spec_dec_mode#

property tokens_per_gen_step: int#: Total tokens per gen request in one spec dec iteration (including golden token).

property uses_external_draft_model: bool#: Whether speculative_model contains an external draft model.

property uses_replacement_heads: bool#: Whether speculative_model contains replacement MTP heads.