config

Configurations for speculative decoding modes.

ModeloptConfig EagleConfig

Bases: ModeloptBaseConfig

Eagle config.

Show default config as JSON

Default config (JSON):

{
   "eagle_num_layers": 1,
   "use_input_layernorm_in_first_layer": true,
   "use_last_layernorm": false
}

field eagle_num_layers: int

Show details

The number of decoder used in the eagle model.

field use_input_layernorm_in_first_layer: bool

Show details

Whether to use input_layernorm in the first decoder layer.

field use_last_layernorm: bool

Show details

Whether to use a final layernorm before lm_head.

ModeloptConfig MedusaConfig

Bases: ModeloptBaseConfig

Medusa config.

Show default config as JSON

Default config (JSON):

{
   "medusa_num_heads": 2,
   "medusa_num_layers": 1
}

field medusa_num_heads: int

Show details

The number of medusa heads added to the model.

field medusa_num_layers: int

Show details

The number of ResBlocks used in medusa head.

ModeloptConfig RedrafterConfig

Bases: ModeloptBaseConfig

Redrafter config.

Show default config as JSON

Default config (JSON):

{
   "redrafter_predict_n_tokens": 2,
   "redrafter_num_layers": 1
}

field redrafter_num_layers: int

Show details

The number of ResBlocks used in lm head.

field redrafter_predict_n_tokens: int

Show details

The number of tokens that redrafter will predict