Installing on Linux via `pip`#

Install TensorRT-LLM (tested on Ubuntu 24.04).

Install prerequisites

Before the pre-built Python wheel can be installed via pip, a few prerequisites must be put into place:

Install CUDA Toolkit following the CUDA Installation Guide for Linux and make sure CUDA_HOME environment variable is properly set.
```
# Optional step: Only required for NVIDIA Blackwell GPUs and SBSA platform
pip3 install torch==2.7.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

sudo apt-get -y install libopenmpi-dev
```
PyTorch CUDA 12.8 package is required for supporting NVIDIA Blackwell GPUs and SBSA platform. On prior GPUs or Linux x86_64 platform, this extra installation is not required.

Tip

Instead of manually installing the preqrequisites as described above, it is also possible to use the pre-built TensorRT-LLM Develop container image hosted on NGC (see here for information on container tags).

Install pre-built TensorRT-LLM wheel

Once all prerequisites are in place, TensorRT-LLM can be installed as follows:
```
pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm
```
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Sanity check the installation by running the following in Python (tested on Python 3.12):

from tensorrt_llm import LLM, SamplingParams


def main():

    # Model could accept HF model name, a path to local HF model,
    # or TensorRT Model Optimizer's quantized checkpoints like nvidia/Llama-3.1-8B-Instruct-FP8 on HF.
    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    # Sample prompts.
    prompts = [
        "Hello, my name is",
        "The capital of France is",
        "The future of AI is",
    ]

    # Create a sampling params.
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

    for output in llm.generate(prompts, sampling_params):
        print(
            f"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}"
        )

    # Got output like
    # Prompt: 'Hello, my name is', Generated text: '\n\nJane Smith. I am a student pursuing my degree in Computer Science at [university]. I enjoy learning new things, especially technology and programming'
    # Prompt: 'The president of the United States is', Generated text: 'likely to nominate a new Supreme Court justice to fill the seat vacated by the death of Antonin Scalia. The Senate should vote to confirm the'
    # Prompt: 'The capital of France is', Generated text: 'Paris.'
    # Prompt: 'The future of AI is', Generated text: 'an exciting time for us. We are constantly researching, developing, and improving our platform to create the most advanced and efficient model available. We are'


if __name__ == '__main__':
    main()

Known limitations

There are some known limitations when you pip install pre-built TensorRT-LLM wheel package.

MPI in the Slurm environment

If you encounter an error while running TensorRT-LLM in a Slurm-managed cluster, you need to reconfigure the MPI installation to work with Slurm. The setup methods depends on your slurm configuration, pls check with your admin. This is not a TensorRT-LLM specific, rather a general mpi+slurm issue.
```
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM support. This usually happens
when OMPI was not configured --with-slurm and we weren't able
to discover a SLURM installation in the usual places.
```

Installing on Linux via pip#

Installing on Linux via `pip`#