Support Matrix

TensorRT-LLM optimizes the performance of a range of well-known models on NVIDIA GPUs. The following sections provide a list of supported GPU architectures as well as important features implemented in TensorRT-LLM.

Models

LLM Models

• Arctic

• Baichuan/Baichuan2

• BART

• BERT

• BLOOM

• ByT5

• GLM/ChatGLM/ChatGLM2/ChatGLM3/GLM-4

• Code LLaMA

• DBRX

• Exaone

• FairSeq NMT

• Falcon

• Flan-T5 [1]

• Gemma/Gemma2

• GPT

• GPT-J

• GPT-Nemo

• GPT-NeoX

• Grok-1

• InternLM

• InternLM2

• LLaMA/LLaMA 2/LLaMA 3/LLaMA 3.1

• Mamba

• mBART

• [Minitron] (https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/nemotron)

• Mistral

• Mistral NeMo

• Mixtral

• MPT

• Nemotron

• mT5

• OPT

• Phi-1.5/Phi-2/Phi-3

• Qwen/Qwen1.5/Qwen2

• Qwen-VL

• RecurrentGemma

• Replit Code[^ReplitCode]

• RoBERTa

• SantaCoder

• Skywork

• Smaug

• StarCoder

• T5

• Whisper

Multi-Modal Models [2]

• BLIP2 w/ OPT

• BLIP2 w/ T5

• CogVLM [3]

• Deplot

• Fuyu

• Kosmos

• LLaVA-v1.5

• LLaVa-Next

• LLaVa-OneVision

• NeVA

• Nougat

• Phi-3-vision

• Video NeVA

• VILA

Hardware

The following table shows the supported hardware for TensorRT-LLM.

If a GPU architecture is not listed, the TensorRT-LLM team does not develop or test the software on the architecture and support is limited to community support. In addition, older architectures can have limitations for newer software releases.

	Hardware Compatibility
Operating System	TensorRT-LLM requires Linux x86_64, Linux aarch64 or Windows.
GPU Model Architectures	NVIDIA Grace Hopper Superchip NVIDIA Hopper Architecture NVIDIA Ada Lovelace Architecture NVIDIA Ampere Architecture

Software

The following table shows the supported software for TensorRT-LLM.

	Software Compatibility
Container	24.10
TensorRT	10.6
Precision	Hopper (SM90) - FP32, FP16, BF16, FP8, INT8, INT4 Ada Lovelace (SM89) - FP32, FP16, BF16, FP8, INT8, INT4 Ampere (SM80, SM86) - FP32, FP16, BF16, INT8, INT4[4]

[^ReplitCode]：Replit Code is not supported with the transformers 4.45+.

Note

Support for FP8 and quantized data types (INT8 or INT4) is not implemented for all the models. Refer to Numerical Precision and examples folder for additional information.

---

[1]

Encoder-Decoder provides general encoder-decoder functionality that supports many encoder-decoder models such as T5 family, BART family, Whisper family, NMT family, and so on.

[2]

Multi-modal provides general multi-modal functionality that supports many multi-modal architectures such as BLIP2 family, LLaVA family, and so on.

[3]

Only supports bfloat16 precision.

[4]

INT4 AWQ and GPTQ with FP8 activations require SM >= 89.