trtllm-serve#

About#

The trtllm-serve command starts an OpenAI compatible server that supports the following endpoints:

/v1/models
/v1/completions
/v1/chat/completions

For information about the inference endpoints, refer to the OpenAI API Reference.

The server also supports the following endpoints:

/health
/metrics
/version
/start_profile (prototype)
/stop_profile (prototype)
/v1/chat/completions/render and /v1/completions/render (prototype, enabled with TRTLLM_ENABLE_RENDER_ENDPOINTS=1)

The metrics endpoint provides runtime-iteration statistics such as GPU memory use and inflight-batching details. The start_profile and stop_profile endpoints control iteration-scoped profiling of the backend engine; see Runtime Profiling Endpoints below. The render endpoints return the prompt token ids that the server would use for a request, without running the model; see Render endpoints.

For encoder-only models (BERT-style classifiers, reward models, text-embedding models), the trtllm-serve embeddings subcommand starts a server that exposes an OpenAI-compatible /v1/embeddings endpoint with native dynamic batching. See Embeddings for details.

Starting a Server#

The following abbreviated command syntax shows the commonly used arguments to start a server:

trtllm-serve <model> [--tp_size <tp> --pp_size <pp> --ep_size <ep> --host <host> --port <port>]

For the full syntax and argument descriptions, refer to Syntax.

Tool-call Parser Selection#

Select --tool_parser to match the tool-call syntax specified by the checkpoint’s chat template. Models from different families can share a parser when they emit the same format.

The qwen3_coder parser handles the following function/parameter format, shared by Qwen3-Coder, Nemotron 3 Nano, Nemotron 3 Super, Nemotron 3 Ultra, and Nemotron 3.5 Super VL:

<tool_call>
<function=get_weather>
<parameter=city>
Taipei
</parameter>
</function>
</tool_call>

The official chat templates specify this syntax and use it to serialize previous assistant tool calls:

Among these Nemotron models, --tool_parser auto selects qwen3_coder only for Nemotron 3.5 Super VL (nemotron_h_omni). For Nemotron 3 Nano, Super, and Ultra (nemotron_h), specify --tool_parser qwen3_coder explicitly.

The module tensorrt_llm/serve/tool_parser/qwen3_coder_parser.py implements this shared format. --reasoning_parser separately controls reasoning extraction; nemotron-v3 and nano-v3 are aliases for the same Nemotron reasoning parser.

Multiple HTTP Frontends#

At high concurrency a single serving process is host-bound: one asyncio event loop parses every request and writes every streamed chunk, and queuing on that loop rather than GPU time starts to dominate first-token latency. trtllm-serve therefore runs 8 HTTP frontend processes against one executor by default (--num_serve_frontends, beta). The launcher process owns the engine; the other frontends attach to it and share the serving port via SO_REUSEPORT, so clients still see a single URL.

Note

Pass --num_serve_frontends 1 (or set num_serve_frontends: 1 in the --config YAML) to run a single frontend.
The default applies to the OpenAI HTTP server on the default executor path on a fixed port. With --port 0 / --report_addr, orchestrator_type (rpc / ray), enable_resource_governor or --enable_sleep_mode it falls back to one frontend and logs why; an explicit --num_serve_frontends greater than 1 fails instead. --grpc also keeps one frontend unless --num_serve_frontends is given (only --grpc-protocol openengine supports more).
Each request is handled entirely by the frontend that accepted its connection. Server-wide state lives in the launcher: the other frontends forward /metrics, /kv_cache_events, profiling and the Anthropic batches API to it, and /prometheus/metrics covers every frontend. Per-process state is not shared: the stateful Responses API store (store / previous_response_id) is disabled, /health describes the frontend that answered, and each frontend writes its own per-request perf metrics file.
SO_REUSEPORT would also let a second server of the same user join the port and take part of its traffic, so the launcher refuses a port that another server already uses.
Every extra frontend is a full Python process that imports TensorRT LLM and loads the tokenizer, so budget host memory accordingly.

Inference Endpoints#

After you start the server, you can send inference requests through completions API, Chat API and Responses API, which are compatible with corresponding OpenAI APIs. We use TinyLlama-1.1B-Chat-v1.0 for examples in the following sections.

Chat API#

You can query Chat API with any http clients, a typical example is OpenAI Python client:

### :title OpenAI Chat Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

response = client.chat.completions.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    messages=[{
        "role": "system",
        "content": "you are a helpful assistant"
    }, {
        "role": "user",
        "content": "Where is New York?"
    }],
    max_tokens=20,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama-1.1B-Chat-v1.0",
        "messages":[{"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": "Where is New York?"}],
        "max_tokens": 16,
        "temperature": 0
    }'

Completions API#

You can query Completions API with any http clients, a typical example is OpenAI Python client:

### :title OpenAI Completion Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

response = client.completions.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    prompt="Where is New York?",
    max_tokens=20,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama-1.1B-Chat-v1.0",
        "prompt": "Where is New York?",
        "max_tokens": 16,
        "temperature": 0
    }'

Responses API#

You can query Responses API with any http clients, a typical example is OpenAI Python client:

### :title OpenAI Responses Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

response = client.responses.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    input="Where is New York?",
    max_output_tokens=20,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

curl http://localhost:8000/v1/responses \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama-1.1B-Chat-v1.0",
        "input": "Where is New York?",
        "max_output_tokens": 16
    }'

More openai compatible examples can be found in the compatibility examples directory.

Multimodal Serving#

For multimodal models, you need to create a configuration file and start the server with additional options due to the following limitations:

TRT-LLM multimodal is currently not compatible with kv_cache_reuse
Multimodal models require chat_template, so only the Chat API and the Responses API are supported

To set up multimodal models:

First, create a configuration file:

cat >./config.yml<<EOF
kv_cache_config:
    enable_block_reuse: false
EOF

Then, start the server with the configuration file:

trtllm-serve Qwen/Qwen2-VL-7B-Instruct \
    --config ./config.yml

Multimodal Chat API#

You can query the Chat Completions API with any HTTP client. A typical example uses the OpenAI Python client:

### :title OpenAI Chat Client for Multimodal

import os
from pathlib import Path

from openai import OpenAI
from PIL import Image

from tensorrt_llm.inputs import (encode_base64_content_from_url,
                                 encode_base64_image)

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

llm_models_root = Path(os.environ.get("LLM_MODELS_ROOT"))

if llm_models_root is not None:
    # These are paths on the server's own filesystem. trtllm-serve refuses
    # local paths unless it is started with
    # TRTLLM_MEDIA_ALLOWED_LOCAL_PATH=$LLM_MODELS_ROOT/multimodals/test_data
    # (or another directory that contains the files).
    multimodal_test_data_path = llm_models_root / "multimodals" / "test_data"
    image_url1 = str(multimodal_test_data_path / "seashore.png")
    image_url2 = str(multimodal_test_data_path / "inpaint.png")
    video_url = str(multimodal_test_data_path / "OAI-sora-tokyo-walk.mp4")
    image64 = encode_base64_image(
        Image.open(multimodal_test_data_path / "seashore.png"))
else:
    image_url1 = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
    image_url2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
    video_url = "https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4"
    image64 = encode_base64_content_from_url(
        "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
    )

# SINGLE IMAGE INFERENCE
response = client.chat.completions.create(
    model="Qwen2.5-VL-3B-Instruct",
    messages=[{
        "role": "system",
        "content": "you are a helpful assistant"
    }, {
        "role":
        "user",
        "content": [{
            "type": "text",
            "text": "Describe the natural environment in the image."
        }, {
            "type": "image_url",
            "image_url": {
                "url": image_url1
            }
        }]
    }],
    max_tokens=64,
)
print(response)

# MULTI IMAGE INFERENCE
response = client.chat.completions.create(
    model="Qwen2.5-VL-3B-Instruct",
    messages=[{
        "role": "system",
        "content": "you are a helpful assistant"
    }, {
        "role":
        "user",
        "content": [{
            "type": "text",
            "text": "Tell me the difference between two images"
        }, {
            "type": "image_url",
            "image_url": {
                "url": image_url2
            }
        }, {
            "type": "image_url",
            "image_url": {
                "url": image_url1
            }
        }]
    }],
    max_tokens=64,
)
print(response)

# SINGLE VIDEO INFERENCE
response = client.chat.completions.create(
    model="Qwen2.5-VL-3B-Instruct",
    messages=[{
        "role": "system",
        "content": "you are a helpful assistant"
    }, {
        "role":
        "user",
        "content": [{
            "type": "text",
            "text": "Tell me what you see in the video briefly."
        }, {
            "type": "video_url",
            "video_url": {
                "url": video_url
            }
        }]
    }],
    max_tokens=64,
)
print(response)

# IMAGE EMBED INFERENCE
response = client.chat.completions.create(
    model="Qwen2.5-VL-3B-Instruct",
    messages=[{
        "role": "system",
        "content": "you are a helpful assistant"
    }, {
        "role":
        "user",
        "content": [{
            "type": "text",
            "text": "Describe the natural environment in the image."
        }, {
            "type": "image_url",
            "image_url": {
                "url": "data:image/png;base64," + image64
            }
        }]
    }],
    max_tokens=64,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

# SINGLE IMAGE INFERENCE
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json"  \
    -d '{
        "model": "Qwen2.5-VL-3B-Instruct",
        "messages":[{
            "role": "system",
            "content": "You are a helpful assistant."
        }, {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe the natural environment in the image."
                },
                {
                    "type":"image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
                    }
                }
            ]
        }],
        "max_tokens": 64,
        "temperature": 0
    }'

# MULTI IMAGE INFERENCE
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2.5-VL-3B-Instruct",
        "messages":[{
            "role": "system",
            "content": "You are a helpful assistant."
        }, {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text":"Tell me the difference between two images"
                },
                {
                    "type":"image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
                    }
                },
                {
                    "type":"image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
                    }
                }
            ]
        }],
        "max_tokens": 64,
        "temperature": 0
    }'

# SINGLE VIDEO INFERENCE
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2.5-VL-3B-Instruct",
        "messages":[{
            "role": "system",
            "content": "You are a helpful assistant."
        }, {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text":"Tell me what you see in the video briefly."
                },
                {
                    "type":"video_url",
                    "video_url": {
                        "url": "https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4"
                    }
                }
            ]
        }],
        "max_tokens": 64,
        "temperature": 0
    }'

Multimodal Responses API#

The Responses API takes an image as an input_image content part whose image_url is the image URL or a base64 data URL:

{"role": "user", "content": [
    {"type": "input_text", "text": "What's in this image?"},
    {"type": "input_image", "image_url": "data:image/jpeg;base64,{image_base64}"}
]}

Images referenced by file_id and input_file content are not supported; a request that carries them is rejected with a 400 error.

Multimodal Modality Coverage#

TRT-LLM multimodal supports the following modalities and data types (depending on the model):

Text

No type specified:

{"role": "user", "content": "What's the capital of South Korea?"}

Explicit “text” type:

{"role": "user", "content": [{"type": "text", "text": "What's the capital of South Korea?"}]}

Image

Using “image_url” with URL:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}
]}

Using “image_url” with base64-encoded data:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,{image_base64}"}}
]}

Note

Media URLs may be http(s) URLs or base64 data: URLs. A file:// URI or a bare filesystem path is read from the server’s own filesystem, so it is refused unless the server is started with TRTLLM_MEDIA_ALLOWED_LOCAL_PATH=<directory>; then only regular files whose resolved path is inside that directory are read. This applies to image_url, video_url and audio_url parts. A deployment whose clients send local paths today must set the variable to the directory that holds those files.

Note

To convert images to base64-encoded format, use the utility function tensorrt_llm.utils.load_base64_image(). Refer to the load_base64_image utility for implementation details.

Image embeddings

It is also possible to directly provide the image embeddings to use by the multimodal model.

Using “image_embeds” with base64-encoded data:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_embeds", "image_embeds": {"data": "{image_embeddings_base64}"}}}
]}

Note

The contents of image_embeddings_base64 can be generated by base64-encoding the result of serializing a tensor via torch.save.

Video

Using “video_url”:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this video?"},
    {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
]}

Audio

Using “audio_url”:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this audio?"},
    {"type": "audio_url", "audio_url": {"url": "https://example.com/audio.mp3"}}
]}

Visual Generation Serving#

trtllm-serve supports diffusion-based visual generation models (FLUX.1, FLUX.2, Wan2.1, Wan2.2) for image and video generation. When a diffusion model directory is provided (detected by the presence of model_index.json), the server automatically launches in visual generation mode with dedicated endpoints.

Note

VisualGen is in beta stage. APIs, supported models, and optimization options are actively evolving and may change in future releases.

# Video generation (Wan)
trtllm-serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
    --visual_gen_args config.yml

# Image generation (FLUX)
trtllm-serve black-forest-labs/FLUX.2-dev \
    --visual_gen_args config.yml

# Video generation (Cosmos3 hybrid checkpoint)
trtllm-serve nvidia/Cosmos3-Nano \
    --enable_visual_gen

For checkpoints that support both LLM and Visual Generation, such as Cosmos3, pass --enable_visual_gen to select the VisualGen runtime when --visual_gen_args is not specified. The --visual_gen_args flag accepts a YAML file that configures quantization, parallelism, and TeaCache. Available visual generation endpoints include /v1/images/generations, /v1/videos, /v1/videos/sync (with /v1/videos/generations kept as a deprecated alias), and video management APIs.

For full details, see the ../../models/visual-generation.md feature documentation. Example client scripts are available in the examples/visual_gen/serve/ directory.

Render Endpoints#

The render endpoints accept the same request body as /v1/chat/completions and /v1/completions and return the prompt token ids that a trtllm-serve server would build from it, including its chat template, tool and reasoning handling, and tokenization. They do not run the model. The response is a rendered request: the token_ids plus the original request body.

Typical uses:

Routing. A KV-cache-aware router or gateway, such as Dynamo’s endpoint picker, calls a render endpoint to get exactly the token ids the server will see. It hashes them to find the server with the longest cached prefix, then forwards the original request to that server.
Skipping rendering on the server. A client can send the ids to a server’s chat or completions endpoint, so the server does not apply the chat template and tokenize again.

The render endpoints are a prototype and are not registered with trtllm-serve by default. Set TRTLLM_ENABLE_RENDER_ENDPOINTS=1 to register them on a trtllm-serve server, or run the standalone, CPU-only trtllm-render next to your servers. See Render endpoints for the response format and for how to send a rendered request to the chat and completions endpoints.

Multi-node Serving with Slurm#

You can deploy DeepSeek-V3 model across two nodes with Slurm and trtllm-serve

echo -e "enable_attention_dp: true\npytorch_backend_config:\n  enable_overlap_scheduler: true" > config.yml

srun -N 2 -w [NODES] \
    --output=benchmark_2node.log \
    --ntasks 16 --ntasks-per-node=8 \
    --mpi=pmix --gres=gpu:8 \
    --container-image=<CONTAINER_IMG> \
    --container-mounts=/workspace:/workspace \
    --container-workdir /workspace \
    bash -c "trtllm-llmapi-launch trtllm-serve deepseek-ai/DeepSeek-V3 --max_batch_size 161 --max_num_tokens 1160 --tp_size 16 --ep_size 4 --kv_cache_free_gpu_memory_fraction 0.95 --config ./config.yml"

See the source code of trtllm-llmapi-launch for more details.

Metrics Endpoint#

Note

The metrics endpoint is in beta. Some fields, such as CPU memory usage, are not yet available.

Enabling enable_iter_perf_stats in the PyTorch backend can slightly impact performance, depending on the serving configuration. Setting iter_perf_stats_interval: N builds the statistics only every N iterations, which reduces this overhead.

The /metrics endpoint provides runtime iteration statistics such as GPU memory usage and KV cache details.

For the default PyTorch backend, iteration statistics logging is enabled by setting the enable_iter_perf_stats field in a YAML file:

# extra_llm_config.yaml
enable_iter_perf_stats: true

Start the server and specify the --config argument with the path to the YAML file:

trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --config config.yaml

After sending at least one inference request to the server, you can fetch runtime iteration statistics by polling the /metrics endpoint. Since the statistics are stored in an internal queue and removed once retrieved, it’s recommended to poll the endpoint shortly after each request and store the results if needed. Each statistic is returned to only one caller, so poll the endpoint from a single client: when several clients poll at once, one of them may receive an empty list.

curl -X GET http://localhost:8000/metrics

Example output:

[
    {
        "gpuMemUsage": 76665782272,
        "iter": 154,
        "iterLatencyMS": 7.00688362121582,
        "kvCacheStats": {
            "allocNewBlocks": 3126,
            "allocTotalBlocks": 3126,
            "cacheHitRate": 0.00128,
            "freeNumBlocks": 101253,
            "maxNumBlocks": 101256,
            "missedBlocks": 3121,
            "reusedBlocks": 4,
            "tokensPerBlock": 32,
            "usedNumBlocks": 3
        },
        "numActiveRequests": 1
        ...
    }
]

Attention-DP statistics#

With attention data parallelism enabled, attentionDpRank identifies the rank that produced each row’s scheduled-work and KV-cache measurements. KV snapshots are captured with the corresponding iteration and include that rank’s capacity, occupancy, allocation and reuse counters, and cache hit rate. An idle rank can retain cached blocks even when it schedules no requests. If a rank did not measure KV statistics for an iteration, its JSON row has kvCacheStats: null; absence does not mean an empty cache. TLLM_METRICS_ALL_RANKS is not required for this attention-DP fanout.

The shared Prometheus collector uses dp_rank on KV-cache and rank-local inflight-batching metrics. It takes the rank from attentionDpRank, falls back to rank for all-rank statistics, and uses "0" for rows without either field. These metric families also have dp_rank="0" in single-rank serving. Request metrics and configuration-info metrics retain their existing labels.

KV gauges retain each rank’s last measurement until another measured snapshot arrives. Used-block counts, utilization, and cache-hit rate can remain stale throughout periods without fresh snapshots, including idle periods. Zero scheduled requests alone does not imply a missing KV measurement.

Cumulative reused/missed-block counters keep independent baselines per rank across absent snapshots. Rows must arrive in producer order for each rank; interleaving different ranks is supported. A decrease in a rank’s cumulative counter emits a warning and updates the baseline without incrementing the counter. This does not make stale or reordered rows safe. Repeated cumulative values add no work. Per-iteration delta counters are incremented directly and must be delivered once per iteration.

Top-level request counts, queue, memory, iteration latency, batch configuration, average decoded tokens, and speculative-decoding measurements remain owned by rank 0 in the collector. They are not sums of rank-local measurements. The separate kvCacheIterationStats deltas retain their producer ownership and are attached only to rank 0 by attention-DP fanout.

When updating dashboards, retain dp_rank to inspect individual ranks or aggregate it explicitly. Used/free block counts can be summed across ranks; the maximum utilization identifies the fullest rank. Do not sum utilization or cache-hit ratios, or multiply rank-0-owned metrics by the number of ranks.

Runtime Profiling Endpoints#

Note

The /start_profile and /stop_profile endpoints are prototype and only apply to the default PyTorch backend (PyExecutor). APIs, parameters, and behaviour may change in future releases.

The server exposes two HTTP endpoints that control iteration-scoped profiling of the backend engine at runtime, without restarting trtllm-serve and without setting TLLM_PROFILE_START_STOP / TLLM_TORCH_PROFILE_TRACE. Under TP/PP > 1, both endpoints broadcast the window to every rank, so each rank writes its own chrome trace (distinguished by a rank-<N> suffix in the filename) and all ranks profile the same iterations in lockstep.

POST /start_profile#

Start a profile window. All fields in the JSON body are optional.

Field	Type	Description
`output_dir`	string	Directory where chrome traces are written. Defaults to `$TLLM_TORCH_PROFILER_DIR` if set, otherwise `/tmp`. One trace file per rank is written as `trtllm-trace-<profile_id>-rank-<N>.json`; `<profile_id>` is a timestamp + short uuid so successive cycles don’t collide.
`num_steps`	int	Number of engine iterations to capture. When set, the engine stops the window automatically and you do not need to call `/stop_profile`. When omitted, profiling runs until `/stop_profile` is called.
`start_step`	int (default `0`)	Iterations to skip after the call before profiling starts. Use to exclude warmup-like transients from the capture.
`activities`	list of string	Subset of `["CPU", "GPU", "CUDA_PROFILER"]` (default `["CPU", "GPU"]`). When `["CUDA_PROFILER"]` is specified, `torch.profiler` is not started and only `cudaProfilerStart` / `cudaProfilerStop` are called — this composes cleanly with `nsys profile -c cudaProfilerApi`.

Response: 200 with {"message": "Profiling started"} on success. 409 with {"success": false, "message": ...} if a profile window is already active or pending (the backend does not silently queue a second window).

POST /stop_profile#

Stop an in-progress profile window and flush the chrome trace to disk. Takes no body. The call blocks until export_chrome_trace() has run on the backend thread, so by the time the handler returns the file is on disk. Call this only if /start_profile was invoked without num_steps; otherwise the window self-terminates.

Example#

# Start a 30-iteration profile window into /tmp/traces.
curl -s -X POST http://127.0.0.1:8000/start_profile \
    -H "Content-Type: application/json" \
    -d '{"output_dir": "/tmp/traces", "num_steps": 30}'

# ... drive load via /v1/completions or /v1/chat/completions ...

# If you started the window without num_steps, stop it:
curl -s -X POST http://127.0.0.1:8000/stop_profile

# Inspect the resulting chrome traces.
ls /tmp/traces/
# trtllm-trace-1778480665-7993d824-rank-0.json
# trtllm-trace-1778480665-7993d824-rank-1.json   (TP=2)

Open the trace files in ui.perfetto.dev or chrome://tracing. Each captured iteration is wrapped in a step[EXTEND bs=N toks=M] or step[DECODE bs=N] user-annotation scope.

With the overlap scheduler enabled, sampling for an iteration runs after the previous batch is updated, so it cannot share the forward scope. That region is emitted as a separate step[...] sample scope. Only the bare step[...] labels mark iteration boundaries; ignore the sample-suffixed ones when counting iterations.

For a full benchmarking + profiling workflow (including multi-cycle capture under steady-state load), see Run benchmarking with trtllm-serve.

Configuring with YAML Files and `--set`#

Pass a YAML file with --config. Explicit CLI flags override conflicting YAML values; otherwise, YAML values override the defaults.

Note

Non-breaking: --config <file.yaml> is the preferred flag for passing a YAML configuration file. Existing workflows using --extra_llm_api_options <file.yaml> continue to work; it is an equivalent alias.

The file configures tensorrt_llm.llmapi.LlmArgs. For example, set a top-level field as follows:

max_batch_size: 8

Use YAML mappings for nested fields:

moe_config:
    backend: CUTLASS

Use repeatable --set PATH=YAML_VALUE options to override individual fields. Paths use exact, case-sensitive LlmArgs field names and dots for nesting:

trtllm-serve MODEL \
    --config base.yaml \
    --video_pruning_rate 0.4 \
    --set multimodal_config.video_pruning_rate=0.6 \
    --set 'cuda_graph_config.batch_sizes=[1, 2, 4]'

Precedence is fixed, regardless of command-line order:

LlmArgs defaults < --config < dedicated CLI flags < --set

Values use YAML syntax and support null, booleans, finite numbers, strings, lists, and string-keyed mappings. Quote arguments containing spaces or shell characters. An assignment replaces exactly the addressed value; assigning a mapping or list replaces it entirely. The last duplicate path wins, while conflicting parent and child paths are rejected.

--set supports public, YAML-serializable fields in TorchLlmArgs, with the following exceptions:

model and backend: use MODEL and --backend.
telemetry_config: use --telemetry or --no-telemetry.
env_overrides and internal_request_auth_key: use the YAML file.
allow_request_chat_template: use --allow_request_chat_template.
disagg_cluster: use --disagg_cluster_uri or the YAML file.
batched_logits_processor, checkpoint_loader, mpi_session, ray_placement_config.placement_groups, speculative_config.drafter, and speculative_config.resource_manager: use the Python LLM API.

Assigning a whole parent mapping does not bypass these exclusions; a mapping that contains one of the excluded descendant paths is rejected.

--set is available only for normal LLM serving. It is unavailable for VisualGen and the embeddings, mm_embedding_serve, disaggregated, and disaggregated_mpi_worker subcommands. Server-only YAML keys, list indexes, escaped dots, and Hydra/OmegaConf interpolation are also unsupported. Replace an enclosing mapping to set free-form keys that cannot be expressed as a path.

Do not pass secrets through --set because arguments may appear in shell history, process listings, or logs. Eligible configuration fields may also be included in usage telemetry; use --no-telemetry to opt out. See Telemetry.

Editor Validation of YAML Files#

In VS Code, install YAML by Red Hat and add this comment to a serving config to enable validation, completion, and hover documentation:

# yaml-language-server: $schema=https://nvidia.github.io/TensorRT-LLM/latest/_static/schemas/trtllm-serve-config.schema.json
max_batch_size: 8

Other YAML language-server clients can use the same schema. The editor downloads the schema; it does not need a local TensorRT-LLM installation or a running server. Replace latest with the docs version matching your installation, once that release publishes schemas. Existing releases are not retroactively updated.

Use trtllm-serve-disagg-config.schema.json for disaggregated --config and trtllm-serve-visual-gen-config.schema.json for --visual_gen_args, in the same schema directory. These describe YAML files, not CLI flags or --set assignments.

View or download the JSON schemas for this documentation version:

To save a schema, right-click its link and select Save link as.

Schemas check field names, nested structure, types, and declarative constraints. Runtime checks still determine model/GPU compatibility, file existence, and cross-field validity. Custom Python validators, free-form mappings, and the deprecated decoding_config have limited static validation; live Python objects are not configurable in YAML.

For offline use, set $schema to a local JSON file (an absolute path or a path relative to your YAML file):

Source checkouts include the schemas in tensorrt_llm/schemas/; no build is needed to use them.
Release containers expose them at /app/tensorrt_llm/schemas/. The editor must be attached to the container, or otherwise able to access those files.
Wheels include the same schemas. To locate them without importing TensorRT-LLM:

python3 -c 'from importlib.metadata import distribution; print(distribution("tensorrt_llm").locate_file("tensorrt_llm/schemas"))'

When changing configuration definitions or bumping the package version in tensorrt_llm/version.py, regenerate and commit the snapshots using a TensorRT-LLM build and dependencies matching the checkout. Each schema’s $id and title include the package version.

python3 scripts/generate_trtllm_serve_schemas.py
python3 scripts/generate_trtllm_serve_schemas.py --check

The CPU schema tests reject stale snapshots. Wheel and HTML docs builds publish the checked-in files without importing configuration classes. --output-dir can be used to generate a separate local preview.

Syntax#

This syntax section lists all command line arguments for trtllm-serve’s subcommands. Some of the arguments are accompanied with a stability tag indicating their development status. Refer to our API Reference for details

trtllm-serve#

Usage

trtllm-serve [OPTIONS] COMMAND [ARGS]...

disaggregated#

Running server in disaggregated mode

Usage

trtllm-serve disaggregated [OPTIONS]

Options

-c, --config, --config_file <config_file>#: beta Path to the disaggregated serving configuration YAML file.

-m, --metadata_server_config_file <metadata_server_config_file>#: prototype Path to metadata server config file

-t, --server_start_timeout <server_start_timeout>#: beta Server start timeout

-r, --request_timeout <request_timeout>#: beta Request timeout

-l, --log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

-s, --schedule_style <schedule_style>#

beta The schedule style for the disaggregated server.

Options:: context_first | generation_first

--metrics-log-interval <metrics_log_interval>#: deprecated [Deprecated] The interval of logging metrics in seconds. This option is not connected to any functionality and will be removed in a future release.

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

--report_addr <report_addr>#: prototype Write the host:port this server actually bound to this file, atomically, once the socket is bound. Lets the config set port 0 and have the launcher read the kernel-assigned port back instead of reserving one up front.

disaggregated_mpi_worker#

Launching disaggregated MPI worker

Usage

trtllm-serve disaggregated_mpi_worker [OPTIONS]

Options

-c, --config, --config_file <config_file>#: beta Path to the disaggregated serving configuration YAML file.

--log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

embeddings#

Run an OpenAI-compatible /v1/embeddings server for encoder-only models.

Coalesces concurrent requests with a dynamic batcher and serves them through the synchronous llm.encode() fast path (no KV cache / sampler / scheduler). Single-GPU only: the command does not expose tensor/pipeline parallelism.

MODEL: model name | HF checkpoint path

Usage

trtllm-serve embeddings [OPTIONS] MODEL

Options

--host <host>#: Hostname of the server.

--port <port>#: Port of the server.

--log_level <log_level>#

The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--max_batch_size <max_batch_size>#: Maximum batch size coalesced into a single encode() call.

--max_num_tokens <max_num_tokens>#: Maximum number of batched input tokens in each encode() call.

--max_queue_delay <max_queue_delay>#: Dynamic-batching hold window in seconds: how long an incoming request waits for others to join its batch before being dispatched (mirrors Triton’s max_queue_delay_microseconds).

--max_queue_size <max_queue_size>#: Maximum number of in-flight queued requests; further requests are rejected with HTTP 429 (mirrors Triton’s max_queue_size).

--trust_remote_code#: Flag for HF transformers.

--config, --extra_llm_api_options <extra_llm_api_options>#: Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file.

--hf_revision, --revision <revision>#: The revision to use for the HuggingFace model (branch name, tag name, or commit id).

--metadata_server_config_file <metadata_server_config_file>#: Path to metadata server config file

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

Arguments

MODEL#: Required argument

mm_embedding_serve#

Running an OpenAI API compatible server

MODEL: model name or Hugging Face checkpoint path

Usage

trtllm-serve mm_embedding_serve [OPTIONS] MODEL

Options

--host <host>#: beta Hostname of the server.

--port <port>#: beta Port of the server.

--log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--max_batch_size <max_batch_size>#: beta Maximum number of requests that the engine can schedule.

--max_num_tokens <max_num_tokens>#: beta Maximum number of batched input tokens after padding is removed in each batch.

--gpus_per_node <gpus_per_node>#: beta Number of GPUs per node. Default to None, and it will be detected automatically.

--trust_remote_code#: beta Flag for HF transformers.

--config, --extra_encoder_options <extra_encoder_options>#: prototype Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file. Prefer –config over –extra_encoder_options.

--hf_revision, --revision <revision>#: beta The revision to use for the HuggingFace model (branch name, tag name, or commit id).

--free_gpu_memory_fraction <free_gpu_memory_fraction>#: beta Free GPU memory fraction reserved for KV Cache, after allocating model weights and buffers.

--tensor_parallel_size, --tp_size <tensor_parallel_size>#: beta Tensor parallelism size.

--metadata_server_config_file <metadata_server_config_file>#: prototype Path to metadata server config file

--allow_request_chat_template#: prototype Allow clients to supply per-request chat_template values. Only enable this for trusted clients.

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

Arguments

MODEL#: Required argument

mooncake_master#

Own a Mooncake pool for as long as this command runs.

Start this before the servers that join the pool. They wait for the manifest, so the order within a job is not delicate, but nothing can hold capacity or serve a lookup until this is up.

Usage

trtllm-serve mooncake_master [OPTIONS]

Options

--rpc_port <rpc_port>#

Port the store clients reach the master on.

Default:: 50051

--metrics_port <metrics_port>#

Prometheus port. Pool occupancy and eviction are read from here or from the master’s log.

Default:: 9004

--eviction_ratio <eviction_ratio>#

Fraction of the pool freed per eviction pass.

Default:: 0.05

--pool_file <pool_file>#: File to publish the pool manifest to once the master answers. Servers name it as mooncake_store.pool: file://<path>, which is how they reach a master whose host the scheduler chose. Removed on exit so a stale address is never dialed. One is written to –run_dir regardless.

--protocol <protocol>#

Transport every participant will use: ‘rdma’ or ‘tcp’. Recorded in the manifest so the servers need not restate it. TCP is for bring-up only; it invalidates performance conclusions.

Default:: 'rdma'

--metadata_server <metadata_server>#

Mooncake metadata service, recorded in the manifest. P2PHANDSHAKE keeps a separate metadata process out of the deployment.

Default:: 'P2PHANDSHAKE'

--namespace <namespace>#

Default key namespace for this pool, recorded in the manifest. A server may override it.

Default:: 'trtllm'

--run_dir <run_dir>#: Where to keep the master’s log and a copy of the manifest. Defaults to a temporary directory.

--binary <binary>#

The master executable to run, looked up on PATH. It ships with the Mooncake runtime that docker/common/install_mooncake.sh installs.

Default:: 'mooncake_master'

--timeout <timeout>#

Seconds to wait for the master to accept connections before giving up on it.

Default:: 60.0

--heartbeat_seconds <heartbeat_seconds>#

Interval between liveness lines. 0 disables them.

Default:: 300

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

mooncake_pool_report#

Report the capacity a run’s pool actually had, and where blocks landed.

Safe to run after the job, since the records outlive the processes that wrote them.

Usage

trtllm-serve mooncake_pool_report [OPTIONS]

Options

--run_dir <run_dir>#: Required Directory holding the pool manifest and the segments/ records each participating rank wrote.

--master_log <master_log>#: The master’s log, for the block-placement table. Defaults to mooncake_master.log in –run_dir.

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

serve#

Running an OpenAI API compatible server

MODEL: model name or Hugging Face checkpoint path

Usage

trtllm-serve serve [OPTIONS] MODEL

Options

--tokenizer <tokenizer>#: beta Path or name of the tokenizer. When using the PyTorch backend, this replaces the default HuggingFace tokenizer.

--custom_tokenizer <custom_tokenizer>#: prototype Custom tokenizer type: alias (e.g., ‘deepseek_v32’) or Python import path (e.g., ‘tensorrt_llm.tokenizer.deepseek_v32.DeepseekV32Tokenizer’).

--post_processor_hook <post_processor_hook>#: prototype Python import path of a user post-processing hook applied after detokenization and before the per-endpoint response formatter (e.g. ‘my_pkg.guardrail.MyPostProcessorHook’). The class must be importable and picklable, take no constructor arguments, and be callable as ‘__call__(chunk) -> verdict’ (see tensorrt_llm.executor.postprocessor_hook). It runs once per output, per streaming chunk, and may rewrite, suppress, or terminate the output; it owns its own per-request state.

--host <host>#: beta Hostname of the server.

--port <port>#: beta Port of the server.

--backend <backend>#

deprecated The backend to use to serve the model. Default is pytorch backend.

Options:: pytorch

--generation-config <generation_config>#

prototype Sampling defaults source. ‘auto’ loads supported values from the model’s generation_config.json; ‘trtllm’ uses TRT-LLM defaults.

Options:: auto | trtllm

--custom_module_dirs <custom_module_dirs>#: prototype Paths to custom module directories to import.

--log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--max_beam_width <max_beam_width>#: beta Maximum number of beams for beam search decoding.

--max_batch_size <max_batch_size>#: beta Maximum number of requests that the engine can schedule.

--max_num_tokens <max_num_tokens>#: beta Maximum number of batched input tokens after padding is removed in each batch.

--max_seq_len <max_seq_len>#: beta Maximum total length of one request, including prompt and outputs. If unspecified, the value is deduced from the model config.

--tensor_parallel_size, --tp_size <tensor_parallel_size>#: beta Tensor parallelism size.

--pipeline_parallel_size, --pp_size <pipeline_parallel_size>#: beta Pipeline parallelism size.

--context_parallel_size, --cp_size <context_parallel_size>#: beta Context parallelism size.

--moe_expert_parallel_size, --ep_size <moe_expert_parallel_size>#: beta expert parallelism size

--moe_cluster_parallel_size, --cluster_size <moe_cluster_parallel_size>#: deprecated [Deprecated] Expert cluster parallelism size. This option is no longer supported and will be removed in a future release.

--gpus_per_node <gpus_per_node>#: beta Number of GPUs per node. Default to None, and it will be detected automatically.

--free_gpu_memory_fraction, --kv_cache_free_gpu_memory_fraction <free_gpu_memory_fraction>#: beta Free GPU memory fraction reserved for KV Cache, after allocating model weights and buffers.

--kv_cache_dtype <kv_cache_dtype>#

prototype KV cache quantization dtype for PyTorch backend. ‘auto’ uses checkpoint/model metadata; explicit values force override.

Options:: auto | fp8 | nvfp4

--num_postprocess_workers <num_postprocess_workers>#: prototype Number of workers to postprocess raw responses to comply with OpenAI protocol.

--num_serve_frontends <num_serve_frontends>#: beta Number of HTTP or OpenEngine frontend processes serving one executor; values > 1 share the serving port via SO_REUSEPORT (classic IPC executor path only). Without an explicit value, –grpc and configurations that support a single frontend only use 1 and log why.

--num_input_processor_workers <num_input_processor_workers>#: prototype Size of the dedicated thread pool that runs the HF input processor (multimodal preprocess) on the chat and completion endpoints.

--num_media_load_workers <num_media_load_workers>#: prototype Size of the dedicated thread pool that decodes media payloads (image / video / audio) for multimodal requests.

--trust_remote_code#: beta Flag for HF transformers.

--hf_revision, --revision <revision>#: beta The revision to use for the HuggingFace model (branch name, tag name, or commit id). Prefer –hf_revision over –revision.

--config, --extra_llm_api_options <extra_llm_api_options>#: prototype Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file, while –set overrides both. Can be specified as either –config or –extra_llm_api_options.

--set <config_overrides>#: prototype Override a canonical LlmArgs path as PATH=YAML_VALUE. Repeatable; applied after –config and dedicated CLI flags. Not supported for startup, telemetry, environment, server-only, or VisualGen settings.

--reasoning_parser <reasoning_parser>#

prototype Specify the parser for reasoning models. Use ‘auto’ to automatically select based on the model.

Options:: auto | minimax_m2_append_think | minimax_m2 | qwen3_5 | qwen3 | deepseek-r1 | glm_moe_dsa | glm47 | glm45 | glm | deepseek_v4 | deepseek_v41 | poolside_v1 | laguna | minimax_m3 | k-exaone | nano-v3 | nemotron-v3 | gemma4 | kimi_k25 | kimi_k2 | kimi_k3

--tool_parser <tool_parser>#

prototype Specify the parser for tool models. Use ‘auto’ to automatically select based on the model.

Options:: auto | qwen3 | qwen3_coder | kimi_k2 | kimi_k3 | deepseek_v3 | deepseek_v31 | deepseek_v32 | deepseek_v4 | deepseek_v41 | gemma4 | glm4 | glm47 | minimax_m2 | minimax_m3 | poolside_v1

--metadata_server_config_file <metadata_server_config_file>#: prototype Path to metadata server config file

--server_role <server_role>#: prototype Server role for disaggregated serving. CONTEXT=prefill (prompt processing), GENERATION=decode (token generation), MM_ENCODER=multimodal encoder, VISUAL_GEN=visual generation. Required when using service registry.

--otlp_traces_endpoint <otlp_traces_endpoint>#: prototype Target URL to which OpenTelemetry traces will be sent.

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

--disagg_cluster_uri <disagg_cluster_uri>#: prototype URI of the disaggregated cluster.

--enable_chunked_prefill#: prototype Enable chunked prefill

--enable_attention_dp#: beta Enable attention data parallel.

--enable_locality_domains#: prototype Enable locality domain execution for supported PyTorch backend ops.

--media_io_kwargs <media_io_kwargs>#: prototype Keyword arguments for media I/O as a JSON string. Keys are modality names (“video”, “image”, “audio”) whose values are dicts of keyword arguments forwarded to the corresponding loader. Example: ‘{“video”: {“extract_audio”: true, “num_frames”: 16}}’ to enable audio extraction from video files.

--video_pruning_rate <video_pruning_rate>#: prototype Pruning rate for video frames in multimodal models. Applied by Efficient Video Sampling (EVS). None disables EVS, values in [0, 1) enable pruning.

--chat_template <chat_template>#: prototype Specify a custom chat template. Can be a file path or one-liner template string

--allow_request_chat_template#: prototype Allow clients to supply per-request chat_template values. Only enable this for trusted clients.

--middleware <middleware>#: prototype FastAPI middleware import path to add to the server app. Can be specified multiple times. Each value must point to either a middleware class or an async HTTP middleware function.

--grpc#: prototype Run the selected gRPC protocol instead of the OpenAI HTTP server.

--grpc-protocol <grpc_protocol>#

prototype Protocol used when –grpc is enabled.

Options:: smg | openengine

--openengine-metrics-port <openengine_metrics_port>#: prototype OpenEngine Prometheus HTTP port (default: 8001). Enabled by return_perf_metrics or perf_metrics_output_dir in the engine configuration.

--openengine-metrics-host <openengine_metrics_host>#: prototype Bind address for OpenEngine Prometheus (default: –host).

--openengine-enable-load-metrics#: prototype Expose per-rank KV occupancy for OpenEngine routing, independently of KV events.

--served_model_name <served_model_name>#: prototype The model name used in the API. If not specified, the model path is used as the model name. This is useful when the model path is long or when you want to expose a custom name to clients.

--enable_visual_gen#: prototype Enable VisualGen runtime for model checkpoints that support both LLM and Visual Generation. Not required if –visual_gen_args specified or the model supports Visual Generation only.

--visual_gen_args <visual_gen_args>#: prototype Path to a YAML file with VisualGen engine args.

--agent_percentage <agent_percentage>#: prototype The percentage of agent requests to schedule. Defaults to 0.0. Should be between 0.0 and 1.0.

--agent_types <agent_types>#: prototype Types of agents to schedule. Now Only Support Open Deep Research agent.

--report_addr <report_addr>#: prototype Write the host:port this server actually bound to this file, atomically, once the socket is bound. Lets –port 0 be used and have the launcher read the kernel-assigned port back instead of reserving one up front.

--enable_sleep_mode#: prototype Enable runtime-memory checkpointing and authenticated HTTP control endpoints.

Arguments

MODEL#: Required argument

Besides the above examples, trtllm-serve is also used as an entrypoint for performance benchmarking. Please refer to Performance Benchmarking with `trtllm-serve <NVIDIA/TensorRT-LLM>` for more details.

trtllm-serve#

About#

Starting a Server#

Tool-call Parser Selection#

Multiple HTTP Frontends#

Inference Endpoints#

Chat API#

Completions API#

Responses API#

Multimodal Serving#

Multimodal Chat API#

Multimodal Responses API#

Multimodal Modality Coverage#

Visual Generation Serving#

Render Endpoints#

Multi-node Serving with Slurm#

Metrics Endpoint#

Attention-DP statistics#

Runtime Profiling Endpoints#

POST /start_profile#

POST /stop_profile#

Example#

Configuring with YAML Files and --set#

Editor Validation of YAML Files#

Syntax#

trtllm-serve#

disaggregated#

disaggregated_mpi_worker#

embeddings#

mm_embedding_serve#

mooncake_master#

mooncake_pool_report#

serve#

Configuring with YAML Files and `--set`#