config

Configurations for speculative decoding modes.

ModeloptConfig EagleConfig

Bases: ModeloptBaseConfig

Eagle config.

Show default config as JSON

Default config (JSON):

{
   "eagle_num_layers": 1
}

field eagle_num_layers: int

Show details

The number of decoder used in the eagle model.

ModeloptConfig MedusaConfig

Bases: ModeloptBaseConfig

Medusa config.

Show default config as JSON

Default config (JSON):

{
   "medusa_num_heads": 2,
   "medusa_num_layers": 1
}

field medusa_num_heads: int

Show details

The number of medusa heads added to the model.

field medusa_num_layers: int

Show details

The number of ResBlocks used in medusa head.

ModeloptConfig RedrafterConfig

Bases: ModeloptBaseConfig

Redrafter config.

Show default config as JSON

Default config (JSON):

{
   "redrafter_predict_n_tokens": 2,
   "redrafter_num_layers": 1
}

field redrafter_num_layers: int

Show details

The number of ResBlocks used in lm head.

field redrafter_predict_n_tokens: int

Show details

The number of tokens that redrafter will predict