Pre-built release container images on NGC

Pre-built TensorRT-LLM releases are made available as container images on NGC. This is likely the simplest way to obtain TensorRT-LLM. Please refer to the documentation in NGC for usage instructions.

Container image tags

In the example shell commands, x.y.z corresponds to the TensorRT-LLM container version to use. If omitted, IMAGE_TAG will default to tensorrt_llm.__version__ (e.g., this documentation was generated from the 1.0.0rc2 source tree). If this does not work, e.g., because a container for the version you are currently working with has not been released yet, you can try using a container published for a previous GitHub pre-release or release (see also NGC Catalog).

Containers can also be built locally, see NVIDIA/TensorRT-LLM for all related options.