Support Matrix

TensorRT-LLM optimizes the performance of a range of well-known models on NVIDIA GPUs. The following sections provide a list of supported GPU architectures as well as important features implemented in TensorRT-LLM.

Hardware

The following table shows the supported hardware for TensorRT-LLM.

If a GPU is not listed, it is important to note that TensorRT-LLM is expected to work on GPUs based on the Volta, Turing, Ampere, Hopper, and Ada Lovelace architectures. Certain limitations may, however, apply.

	Hardware Compatibility
Operating System	TensorRT-LLM requires Linux x86_64 or Windows.
GPU Model Architectures	NVIDIA Hopper H100 GPU NVIDIA L40S GPU NVIDIA Ada Lovelace GPU NVIDIA Ampere A100 GPU NVIDIA A30 GPU NVIDIA Turing T4 GPU NVIDIA Volta V100 GPU (experimental)

Software

The following table shows the supported software for TensorRT-LLM.

	Software Compatibility
Container	[23.10](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html#framework-matrix-2023)
TensorRT	[9.2](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html)
Precision	Hopper (SM90) - FP32, FP16, BF16, FP8, INT8, INT4 Ada Lovelace (SM89) - FP32, FP16, BF16, FP8, INT8, INT4 Ampere (SM80, SM86) - FP32, FP16, BF16, INT8, INT4(3) Turing (SM75) - FP32, FP16, INT8(1), INT4(2) Volta (SM70) - FP32, FP16, INT8(1), INT4(2)
Models	[Baichuan](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/baichuan) [BART](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/enc_dec) [BERT](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/bert) [Blip2](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/multimodal) [BLOOM](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/bloom) [ChatGLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/chatglm) [DBRX](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/dbrx) [FairSeq NMT](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/enc_dec) [Falcon](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/falcon) [Flan-T5](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/enc_dec) (4) [Gemma](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gemma) [GPT](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gpt) [GPT-J](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gptj) [GPT-Nemo](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gpt) [GPT-NeoX](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gptneox) [InternLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/internlm) [LLaMA](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama) [LLaMA-v2](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama) [Mamba](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/mamba) [mBART](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/enc_dec) [Mistral](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/mixtral) [MPT](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/mpt) [mT5](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/enc_dec) [OPT](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/opt) [Phi-1.5/Phi-2](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/phi) [Qwen](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/qwen) [Qwen-VL](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/qwenvl) [Replit Code](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/mpt) [RoBERTa](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/bert) [SantaCoder](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gpt) [Skywork](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/skywork) [Smaug](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/smaug) [StarCoder](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/gpt) [T5](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/enc_dec) [Whisper](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/whisper)
Multi-Modal Models (5)	[BLIP2 w/ OPT-2.7B](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/multimodal) [BLIP2 w/ T5-XL](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/multimodal) [LLaVA-v1.5-7B](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/multimodal) [Nougat family](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/multimodal) Nougat-small, Nougat-base

(1) INT8 SmoothQuant is not supported on SM70 and SM75.
(2) INT4 AWQ and GPTQ are not supported on SM < 80.
(3) INT4 AWQ and GPTQ with FP8 activations require SM >= 89.
(4) Encoder-Decoder provides general encoder-decoder functionality that supports many encoder-decoder models such as T5 family, BART family, Whisper family, NMT family, and so on. (5) Multi-modal provides general multi-modal functionality that supports many multi-modal architectures such as BLIP family, LLaVA family, and so on.

Note

Support for FP8 and quantized data types (INT8 or INT4) is not implemented for all the models. Refer to Numerical Precision and examples folder for additional information.