Installing on Grace Hopper

  1. Install TensorRT-LLM (tested on Ubuntu 24.04).

    pip3 install torch==2.5.1 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

sudo apt-get -y install libopenmpi-dev && pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm --extra-index-url https://pypi.nvidia.com

    If using the PyTorch NGC Container image, the prerequisite step for installing CUDA-enabled PyTorch package is not required.

  2. Sanity check the installation by running the following in Python (tested on Python 3.12):

     1from tensorrt_llm import LLM, SamplingParams
 2
 3
 4def main():
 5
 6    prompts = [
 7        "Hello, my name is",
 8        "The president of the United States is",
 9        "The capital of France is",
10        "The future of AI is",
11    ]
12    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
13
14    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
15
16    outputs = llm.generate(prompts, sampling_params)
17
18    # Print the outputs.
19    for output in outputs:
20        prompt = output.prompt
21        generated_text = output.outputs[0].text
22        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
23
24
25# The entry point of the program need to be protected for spawning processes.
26if __name__ == '__main__':
27    main()