config

Configurations for speculative decoding modes.

ModeloptConfig EagleConfig

Bases: ModeloptBaseConfig

Eagle config.

Show default config as JSON

Default config (JSON):

{
   "eagle_offline": false,
   "eagle_hidden_state_distillation": false,
   "eagle_self_logit_distillation": true,
   "eagle_freeze_base_model": true,
   "eagle_report_acc": true,
   "eagle_reuse_base_decoder": false,
   "eagle_loss_decay_factor": 0.9,
   "eagle_architecture_config": {}
}

field eagle_architecture_config: dict

Show details

The config for eagle module architecture.

field eagle_freeze_base_model: bool

Show details

Whether to freeze base model during eagle module training.

field eagle_hidden_state_distillation: bool

Show details

Whether to use feature hidden states distillation.

field eagle_loss_decay_factor: float

Show details

The decay factor for multiple eagle_loss.

field eagle_offline: bool

Show details

Whether to use detached Eagle.

field eagle_report_acc: bool

Show details

Whether to report eval accuracy.

field eagle_reuse_base_decoder: bool

Show details

Whether to reuse base model decoder in eagle module.

field eagle_self_logit_distillation: bool

Show details

Whether to use logit distillation.

ModeloptConfig MedusaConfig

Bases: ModeloptBaseConfig

Medusa config.

Show default config as JSON

Default config (JSON):

{
   "medusa_num_heads": 2,
   "medusa_num_layers": 1
}

field medusa_num_heads: int

Show details

The number of medusa heads added to the model.

field medusa_num_layers: int

Show details

The number of ResBlocks used in medusa head.