config
Configurations for speculative decoding modes.
- ModeloptConfig EagleConfig
Bases:
ModeloptBaseConfigEagle config.
Show default config as JSON
- Default config (JSON):
{ "eagle_offline": false, "eagle_hidden_state_distillation": false, "eagle_self_logit_distillation": true, "eagle_freeze_base_model": true, "eagle_report_acc": true, "eagle_reuse_base_decoder": false, "eagle_loss_decay_factor": 0.9, "eagle_architecture_config": {}, "eagle_decoder_type": "llama", "eagle_ttt_steps": 3, "eagle_mix_hidden_states": false, "eagle_use_torch_compile": true, "eagle_enable_nvtx": false }
- field eagle_architecture_config: dict
Show details
The config for eagle module architecture.
- field eagle_decoder_type: str
Show details
The class of eagle decoder to use. Available options: llama, kimik2
- field eagle_enable_nvtx: bool
Show details
Whether to enable NVTX ranges for profiling eagle forward/loss methods.
- field eagle_freeze_base_model: bool
Show details
Whether to freeze base model during eagle module training.
Show details
Whether to use feature hidden states distillation.
- field eagle_loss_decay_factor: float
Show details
The decay factor for multiple eagle_loss.
Show details
Whether to mix hidden states of multiple TTT steps. It is a technique to reduce training cost.
- field eagle_offline: bool
Show details
Whether to use detached Eagle.
- field eagle_report_acc: bool
Show details
Whether to report eval accuracy.
- field eagle_reuse_base_decoder: bool
Show details
Whether to reuse base model decoder in eagle module.
- field eagle_self_logit_distillation: bool
Show details
Whether to use logit distillation.
- field eagle_ttt_steps: int
Show details
The number of train-time-test steps in training.
- field eagle_use_torch_compile: bool
Show details
Whether to use torch.compile on eagle forward/loss methods for faster training.
- ModeloptConfig MedusaConfig
Bases:
ModeloptBaseConfigMedusa config.
Show default config as JSON
- Default config (JSON):
{ "medusa_num_heads": 2, "medusa_num_layers": 1 }
- field medusa_num_heads: int
Show details
The number of medusa heads added to the model.
- field medusa_num_layers: int
Show details
The number of ResBlocks used in medusa head.