Quantization

class tensorrt_llm.quantization.QuantAlgo(value)[source]

Bases: StrEnum

An enumeration.

class tensorrt_llm.quantization.QuantMode(value)[source]

Bases: IntFlag

An enumeration.

tensorrt_llm.quantization.quantize_and_export(*, model_dir, device, calib_dataset, dtype, qformat, kv_cache_dtype, calib_size, batch_size, calib_max_seq_length, awq_block_size, output_dir, tp_size, pp_size, seed, tokenizer_max_seq_length, num_medusa_heads=None, num_medusa_layers=None, max_draft_len=None, medusa_hidden_act=None, medusa_model_dir=None, quant_medusa_head=None, auto_quantize_bits=None)[source]: Load model from the model_dir, call Modelopt to quantize the model, and then export the quantized model as TRT-LLM checkpoint