trtedge CLI#

trtedge is the command-line interface (CLI) for chat, HTTP serving, engine building, ONNX export, and model preparation.

trtedge COMMAND [OPTIONS]

Complete Installation and activate your Python environment before using trtedge. For a first run, follow Quick Start.

Flag spelling#

Use hyphenated long options, such as --max-input-len and --model-dir. The standalone C++ tools also accept their existing camelCase spellings (for example, llm_build --maxInputLen), and Python model-preparation commands retain underscore aliases (for example, trtedge quantize llm --model_dir). Existing scripts can continue using those spellings. Configuration-file keys and Python API parameter names are unchanged.

Commands#

The recommended installation includes [server] for chat and HTTP serving. ONNX export requires [export], and quantization requires [tools]. See Optional Python dependencies to add these extras to your environment.

Command	Purpose	Guide
`trtedge chat`	Interactive or one-shot text chat	Interactive Text Chat
`trtedge serve`	Launch an OpenAI-compatible HTTP server	Python API and Server
`trtedge build`	Build engines directly from a checkpoint	Direct Engine Builder
`trtedge export`	Export checkpoint components to ONNX	ONNX export example
`trtedge quantize`	Quantize an LLM or draft checkpoint	Quantization
`trtedge insert-lora`	Insert LoRA patterns into ONNX	LoRA
`trtedge process-lora`	Process LoRA adapter weights	LoRA
`trtedge merge-lora`	Merge a LoRA adapter into a checkpoint	LoRA
`trtedge reduce-vocab`	Generate reduced-vocabulary artifacts	Vocabulary Reduction

Examples#

Chat with a model#

Start an interactive conversation with a Hugging Face model or a local checkpoint:

trtedge chat Qwen/Qwen3.5-0.8B
trtedge chat /path/to/checkpoint

To run one prompt:

trtedge chat Qwen/Qwen3.5-0.8B --prompt "Explain CUDA streams briefly."

Chat and serving build and cache TensorRT engines on first use. Subsequent launches reuse matching cached engines.

Launch a server#

trtedge serve Qwen/Qwen3.5-0.8B --port 8000

See the server guide for HTTP requests, speculative decoding, and runtime options.

Build engines or export ONNX#

Build TensorRT engines directly from a local checkpoint:

trtedge build --model-dir /path/to/checkpoint --engine-dir /path/to/engines

Export ONNX for the component-specific C++ builders:

trtedge export /path/to/checkpoint /path/to/onnx-output

Quantize a checkpoint#

trtedge quantize llm --model-dir /path/to/checkpoint \
  --output-dir /path/to/quantized --quantization fp8

Choose a quantization format supported by your model and target device; see Quantization.

Help and version#

trtedge --help
trtedge chat --help
trtedge --version

Use trtedge COMMAND --help for command arguments and options.

Standalone command names#

You can also use the following standalone commands. Each accepts the same arguments and options as its corresponding trtedge subcommand.

`trtedge` command	Standalone command
`trtedge serve`	`tensorrt-edgellm-serve`
`trtedge build`	`tensorrt-edgellm-build`
`trtedge export`	`tensorrt-edgellm-export`
`trtedge quantize`	`tensorrt-edgellm-quantize`
`trtedge insert-lora`	`tensorrt-edgellm-insert-lora`
`trtedge process-lora`	`tensorrt-edgellm-process-lora`
`trtedge merge-lora`	`tensorrt-edgellm-merge-lora`
`trtedge reduce-vocab`	`tensorrt-edgellm-reduce-vocab`