vllm

Utility to convert a Model Optimizer exported model to vLLM Checkpoint.

Functions

export_to_vllm

Exports the torch model to vLLM checkpoint and saves to export_dir.

export_to_vllm(model, tokenizer, export_path='/tmp')

Exports the torch model to vLLM checkpoint and saves to export_dir.

Parameters:

• model (Module) – the torch model

• tokenizer (Module) – the tokenizer used for model

• export_path (Path | str) – Path for exporting the vLLM compatible quantized checkpoint