trtllm-serve#

About#

The trtllm-serve command starts an OpenAI compatible server that supports the following endpoints:

/v1/models
/v1/completions
/v1/chat/completions

For information about the inference endpoints, refer to the OpenAI API Reference.

The server also supports the following endpoints:

/health
/metrics
/version
/start_profile (prototype)
/stop_profile (prototype)

The metrics endpoint provides runtime-iteration statistics such as GPU memory use and inflight-batching details. The start_profile and stop_profile endpoints control iteration-scoped profiling of the backend engine; see Runtime Profiling Endpoints below.

For encoder-only models (BERT-style classifiers, reward models, text-embedding models), the trtllm-serve embeddings subcommand starts a server that exposes an OpenAI-compatible /v1/embeddings endpoint with native dynamic batching. See Embeddings for details.

Starting a Server#

The following abbreviated command syntax shows the commonly used arguments to start a server:

trtllm-serve <model> [--tp_size <tp> --pp_size <pp> --ep_size <ep> --host <host> --port <port>]

For the full syntax and argument descriptions, refer to Syntax.

Inference Endpoints#

After you start the server, you can send inference requests through completions API, Chat API and Responses API, which are compatible with corresponding OpenAI APIs. We use TinyLlama-1.1B-Chat-v1.0 for examples in the following sections.

Chat API#

You can query Chat API with any http clients, a typical example is OpenAI Python client:

### :title OpenAI Chat Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

response = client.chat.completions.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    messages=[{
        "role": "system",
        "content": "you are a helpful assistant"
    }, {
        "role": "user",
        "content": "Where is New York?"
    }],
    max_tokens=20,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama-1.1B-Chat-v1.0",
        "messages":[{"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": "Where is New York?"}],
        "max_tokens": 16,
        "temperature": 0
    }'

Completions API#

You can query Completions API with any http clients, a typical example is OpenAI Python client:

### :title OpenAI Completion Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

response = client.completions.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    prompt="Where is New York?",
    max_tokens=20,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama-1.1B-Chat-v1.0",
        "prompt": "Where is New York?",
        "max_tokens": 16,
        "temperature": 0
    }'

Responses API#

You can query Responses API with any http clients, a typical example is OpenAI Python client:

### :title OpenAI Responses Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="tensorrt_llm",
)

response = client.responses.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    input="Where is New York?",
    max_output_tokens=20,
)
print(response)

Another example uses curl:

#! /usr/bin/env bash

curl http://localhost:8000/v1/responses \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama-1.1B-Chat-v1.0",
        "input": "Where is New York?",
        "max_output_tokens": 16
    }'

More openai compatible examples can be found in the compatibility examples directory.

Multimodal Serving#

For multimodal models, you need to create a configuration file and start the server with additional options due to the following limitations:

TRT-LLM multimodal is currently not compatible with kv_cache_reuse
Multimodal models require chat_template, so only the Chat API is supported

To set up multimodal models:

First, create a configuration file:

cat >./config.yml<<EOF
kv_cache_config:
    enable_block_reuse: false
EOF

Then, start the server with the configuration file:

trtllm-serve Qwen/Qwen2-VL-7B-Instruct \
    --config ./config.yml

Multimodal Chat API#

You can query Completions API with any http clients, a typical example is OpenAI Python client:

Another example uses curl:

#! /usr/bin/env bash

# SINGLE IMAGE INFERENCE
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json"  \
    -d '{
        "model": "Qwen2.5-VL-3B-Instruct",
        "messages":[{
            "role": "system",
            "content": "You are a helpful assistant."
        }, {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe the natural environment in the image."
                },
                {
                    "type":"image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
                    }
                }
            ]
        }],
        "max_tokens": 64,
        "temperature": 0
    }'

# MULTI IMAGE INFERENCE
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2.5-VL-3B-Instruct",
        "messages":[{
            "role": "system",
            "content": "You are a helpful assistant."
        }, {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text":"Tell me the difference between two images"
                },
                {
                    "type":"image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"
                    }
                },
                {
                    "type":"image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png"
                    }
                }
            ]
        }],
        "max_tokens": 64,
        "temperature": 0
    }'

# SINGLE VIDEO INFERENCE
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen2.5-VL-3B-Instruct",
        "messages":[{
            "role": "system",
            "content": "You are a helpful assistant."
        }, {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text":"Tell me what you see in the video briefly."
                },
                {
                    "type":"video_url",
                    "video_url": {
                        "url": "https://huggingface.co/datasets/Efficient-Large-Model/VILA-inference-demos/resolve/main/OAI-sora-tokyo-walk.mp4"
                    }
                }
            ]
        }],
        "max_tokens": 64,
        "temperature": 0
    }'

Multimodal Modality Coverage#

TRT-LLM multimodal supports the following modalities and data types (depending on the model):

Text

No type specified:

{"role": "user", "content": "What's the capital of South Korea?"}

Explicit “text” type:

{"role": "user", "content": [{"type": "text", "text": "What's the capital of South Korea?"}]}

Image

Using “image_url” with URL:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}
]}

Using “image_url” with base64-encoded data:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,{image_base64}"}}
]}

Note

To convert images to base64-encoded format, use the utility function tensorrt_llm.utils.load_base64_image(). Refer to the load_base64_image utility for implementation details.

Image embeddings

It is also possible to directly provide the image embeddings to use by the multimodal model.

Using “image_embeds” with base64-encoded data:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_embeds", "image_embeds": {"data": "{image_embeddings_base64}"}}}
]}

Note

The contents of image_embeddings_base64 can be generated by base64-encoding the result of serializing a tensor via torch.save.

Video

Using “video_url”:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this video?"},
    {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
]}

Audio

Using “audio_url”:

{"role": "user", "content": [
    {"type": "text", "text": "What's in this audio?"},
    {"type": "audio_url", "audio_url": {"url": "https://example.com/audio.mp3"}}
]}

Visual Generation Serving#

trtllm-serve supports diffusion-based visual generation models (FLUX.1, FLUX.2, Wan2.1, Wan2.2) for image and video generation. When a diffusion model directory is provided (detected by the presence of model_index.json), the server automatically launches in visual generation mode with dedicated endpoints.

Note

VisualGen is in beta stage. APIs, supported models, and optimization options are actively evolving and may change in future releases.

# Video generation (Wan)
trtllm-serve Wan-AI/Wan2.2-T2V-A14B-Diffusers \
    --visual_gen_args config.yml

# Image generation (FLUX)
trtllm-serve black-forest-labs/FLUX.2-dev \
    --visual_gen_args config.yml

# Video generation (Cosmos3 hybrid checkpoint)
trtllm-serve nvidia/Cosmos3-Nano \
    --enable_visual_gen

For checkpoints that support both LLM and Visual Generation, such as Cosmos3, pass --enable_visual_gen to select the VisualGen runtime when --visual_gen_args is not specified. The --visual_gen_args flag accepts a YAML file that configures quantization, parallelism, and TeaCache. Available visual generation endpoints include /v1/images/generations, /v1/videos, /v1/videos/sync (with /v1/videos/generations kept as a deprecated alias), and video management APIs.

For full details, see the ../../models/visual-generation.md feature documentation. Example client scripts are available in the examples/visual_gen/serve/ directory.

Multi-node Serving with Slurm#

You can deploy DeepSeek-V3 model across two nodes with Slurm and trtllm-serve

echo -e "enable_attention_dp: true\npytorch_backend_config:\n  enable_overlap_scheduler: true" > config.yml

srun -N 2 -w [NODES] \
    --output=benchmark_2node.log \
    --ntasks 16 --ntasks-per-node=8 \
    --mpi=pmix --gres=gpu:8 \
    --container-image=<CONTAINER_IMG> \
    --container-mounts=/workspace:/workspace \
    --container-workdir /workspace \
    bash -c "trtllm-llmapi-launch trtllm-serve deepseek-ai/DeepSeek-V3 --max_batch_size 161 --max_num_tokens 1160 --tp_size 16 --ep_size 4 --kv_cache_free_gpu_memory_fraction 0.95 --config ./config.yml"

See the source code of trtllm-llmapi-launch for more details.

Metrics Endpoint#

Note

The metrics endpoint is in beta. Some fields, such as CPU memory usage, are not yet available.

Enabling enable_iter_perf_stats in the PyTorch backend can slightly impact performance, depending on the serving configuration. Setting iter_perf_stats_interval: N builds the statistics only every N iterations, which reduces this overhead.

The /metrics endpoint provides runtime iteration statistics such as GPU memory usage and KV cache details.

For the default PyTorch backend, iteration statistics logging is enabled by setting the enable_iter_perf_stats field in a YAML file:

# extra_llm_config.yaml
enable_iter_perf_stats: true

Start the server and specify the --config argument with the path to the YAML file:

trtllm-serve "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --config config.yaml

After sending at least one inference request to the server, you can fetch runtime iteration statistics by polling the /metrics endpoint. Since the statistics are stored in an internal queue and removed once retrieved, it’s recommended to poll the endpoint shortly after each request and store the results if needed.

curl -X GET http://localhost:8000/metrics

Example output:

[
    {
        "gpuMemUsage": 76665782272,
        "iter": 154,
        "iterLatencyMS": 7.00688362121582,
        "kvCacheStats": {
            "allocNewBlocks": 3126,
            "allocTotalBlocks": 3126,
            "cacheHitRate": 0.00128,
            "freeNumBlocks": 101253,
            "maxNumBlocks": 101256,
            "missedBlocks": 3121,
            "reusedBlocks": 4,
            "tokensPerBlock": 32,
            "usedNumBlocks": 3
        },
        "numActiveRequests": 1
        ...
    }
]

Runtime Profiling Endpoints#

Note

The /start_profile and /stop_profile endpoints are prototype and only apply to the default PyTorch backend (PyExecutor). APIs, parameters, and behaviour may change in future releases.

The server exposes two HTTP endpoints that control iteration-scoped profiling of the backend engine at runtime, without restarting trtllm-serve and without setting TLLM_PROFILE_START_STOP / TLLM_TORCH_PROFILE_TRACE. Under TP/PP > 1, both endpoints broadcast the window to every rank, so each rank writes its own chrome trace (distinguished by a rank-<N> suffix in the filename) and all ranks profile the same iterations in lockstep.

POST /start_profile#

Start a profile window. All fields in the JSON body are optional.

Field	Type	Description
`output_dir`	string	Directory where chrome traces are written. Defaults to `$TLLM_TORCH_PROFILER_DIR` if set, otherwise `/tmp`. One trace file per rank is written as `trtllm-trace-<profile_id>-rank-<N>.json`; `<profile_id>` is a timestamp + short uuid so successive cycles don’t collide.
`num_steps`	int	Number of engine iterations to capture. When set, the engine stops the window automatically and you do not need to call `/stop_profile`. When omitted, profiling runs until `/stop_profile` is called.
`start_step`	int (default `0`)	Iterations to skip after the call before profiling starts. Use to exclude warmup-like transients from the capture.
`activities`	list of string	Subset of `["CPU", "GPU", "CUDA_PROFILER"]` (default `["CPU", "GPU"]`). When `["CUDA_PROFILER"]` is specified, `torch.profiler` is not started and only `cudaProfilerStart` / `cudaProfilerStop` are called — this composes cleanly with `nsys profile -c cudaProfilerApi`.

Response: 200 with {"message": "Profiling started"} on success. 409 with {"success": false, "message": ...} if a profile window is already active or pending (the backend does not silently queue a second window).

POST /stop_profile#

Stop an in-progress profile window and flush the chrome trace to disk. Takes no body. The call blocks until export_chrome_trace() has run on the backend thread, so by the time the handler returns the file is on disk. Call this only if /start_profile was invoked without num_steps; otherwise the window self-terminates.

Example#

# Start a 30-iteration profile window into /tmp/traces.
curl -s -X POST http://127.0.0.1:8000/start_profile \
    -H "Content-Type: application/json" \
    -d '{"output_dir": "/tmp/traces", "num_steps": 30}'

# ... drive load via /v1/completions or /v1/chat/completions ...

# If you started the window without num_steps, stop it:
curl -s -X POST http://127.0.0.1:8000/stop_profile

# Inspect the resulting chrome traces.
ls /tmp/traces/
# trtllm-trace-1778480665-7993d824-rank-0.json
# trtllm-trace-1778480665-7993d824-rank-1.json   (TP=2)

Open the trace files in ui.perfetto.dev or chrome://tracing. Each captured iteration is wrapped in a step[EXTEND bs=N toks=M] or step[DECODE bs=N] user-annotation scope.

With the overlap scheduler enabled, sampling for an iteration runs after the previous batch is updated, so it cannot share the forward scope. That region is emitted as a separate step[...] sample scope. Only the bare step[...] labels mark iteration boundaries; ignore the sample-suffixed ones when counting iterations.

For a full benchmarking + profiling workflow (including multi-cycle capture under steady-state load), see Run benchmarking with trtllm-serve.

Configuring with YAML Files and `--set`#

Pass a YAML file with --config. Explicit CLI flags override conflicting YAML values; otherwise, YAML values override the defaults.

Note

Non-breaking: --config <file.yaml> is the preferred flag for passing a YAML configuration file. Existing workflows using --extra_llm_api_options <file.yaml> continue to work; it is an equivalent alias.

The file configures tensorrt_llm.llmapi.LlmArgs. For example, set a top-level field as follows:

max_batch_size: 8

Use YAML mappings for nested fields:

moe_config:
    backend: CUTLASS

Use repeatable --set PATH=YAML_VALUE options to override individual fields. Paths use exact, case-sensitive LlmArgs field names and dots for nesting:

trtllm-serve MODEL \
    --config base.yaml \
    --video_pruning_rate 0.4 \
    --set multimodal_config.video_pruning_rate=0.6 \
    --set 'cuda_graph_config.batch_sizes=[1, 2, 4]'

Precedence is fixed, regardless of command-line order:

LlmArgs defaults < --config < dedicated CLI flags < --set

Values use YAML syntax and support null, booleans, finite numbers, strings, lists, and string-keyed mappings. Quote arguments containing spaces or shell characters. An assignment replaces exactly the addressed value; assigning a mapping or list replaces it entirely. The last duplicate path wins, while conflicting parent and child paths are rejected.

--set supports public, YAML-serializable fields in TorchLlmArgs, with the following exceptions:

model and backend: use MODEL and --backend.
telemetry_config: use --telemetry or --no-telemetry.
env_overrides and internal_request_auth_key: use the YAML file.
allow_request_chat_template: use --allow_request_chat_template.
disagg_cluster: use --disagg_cluster_uri or the YAML file.
batched_logits_processor, checkpoint_loader, mpi_session, ray_placement_config.placement_groups, speculative_config.drafter, and speculative_config.resource_manager: use the Python LLM API.

Assigning a whole parent mapping does not bypass these exclusions; a mapping that contains one of the excluded descendant paths is rejected.

--set is available only for normal LLM serving. It is unavailable for VisualGen and the embeddings, mm_embedding_serve, disaggregated, and disaggregated_mpi_worker subcommands. Server-only YAML keys, list indexes, escaped dots, and Hydra/OmegaConf interpolation are also unsupported. Replace an enclosing mapping to set free-form keys that cannot be expressed as a path.

Do not pass secrets through --set because arguments may appear in shell history, process listings, or logs. Eligible configuration fields may also be included in usage telemetry; use --no-telemetry to opt out. See Telemetry.

Editor Validation of YAML Files#

In VS Code, install YAML by Red Hat and add this comment to a serving config to enable validation, completion, and hover documentation:

# yaml-language-server: $schema=https://nvidia.github.io/TensorRT-LLM/latest/_static/schemas/trtllm-serve-config.schema.json
max_batch_size: 8

Other YAML language-server clients can use the same schema. The editor downloads the schema; it does not need a local TensorRT-LLM installation or a running server. Replace latest with the docs version matching your installation, once that release publishes schemas. Existing releases are not retroactively updated.

Use trtllm-serve-disagg-config.schema.json for disaggregated --config and trtllm-serve-visual-gen-config.schema.json for --visual_gen_args, in the same schema directory. These describe YAML files, not CLI flags or --set assignments.

View or download the JSON schemas for this documentation version:

To save a schema, right-click its link and select Save link as.

Schemas check field names, nested structure, types, and declarative constraints. Runtime checks still determine model/GPU compatibility, file existence, and cross-field validity. Custom Python validators, free-form mappings, and the deprecated decoding_config have limited static validation; live Python objects are not configurable in YAML.

For offline use, set $schema to a local JSON file (an absolute path or a path relative to your YAML file):

Source checkouts include the schemas in tensorrt_llm/schemas/; no build is needed to use them.
Release containers expose them at /app/tensorrt_llm/schemas/. The editor must be attached to the container, or otherwise able to access those files.
Wheels include the same schemas. To locate them without importing TensorRT-LLM:

python3 -c 'from importlib.metadata import distribution; print(distribution("tensorrt_llm").locate_file("tensorrt_llm/schemas"))'

When changing configuration definitions or bumping the package version in tensorrt_llm/version.py, regenerate and commit the snapshots using a TensorRT-LLM build and dependencies matching the checkout. Each schema’s $id and title include the package version.

python3 scripts/generate_trtllm_serve_schemas.py
python3 scripts/generate_trtllm_serve_schemas.py --check

The CPU schema tests reject stale snapshots. Wheel and HTML docs builds publish the checked-in files without importing configuration classes. --output-dir can be used to generate a separate local preview.

Syntax#

This syntax section lists all command line arguments for trtllm-serve’s subcommands. Some of the arguments are accompanied with a stability tag indicating their development status. Refer to our API Reference for details

trtllm-serve#

Usage

trtllm-serve [OPTIONS] COMMAND [ARGS]...

disaggregated#

Running server in disaggregated mode

Usage

trtllm-serve disaggregated [OPTIONS]

Options

-c, --config, --config_file <config_file>#: beta Path to the disaggregated serving configuration YAML file.

-m, --metadata_server_config_file <metadata_server_config_file>#: prototype Path to metadata server config file

-t, --server_start_timeout <server_start_timeout>#: beta Server start timeout

-r, --request_timeout <request_timeout>#: beta Request timeout

-l, --log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

-s, --schedule_style <schedule_style>#

beta The schedule style for the disaggregated server.

Options:: context_first | generation_first

--metrics-log-interval <metrics_log_interval>#: deprecated [Deprecated] The interval of logging metrics in seconds. This option is not connected to any functionality and will be removed in a future release.

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

--report_addr <report_addr>#: prototype Write the host:port this server actually bound to this file, atomically, once the socket is bound. Lets the config set port 0 and have the launcher read the kernel-assigned port back instead of reserving one up front.

disaggregated_mpi_worker#

Launching disaggregated MPI worker

Usage

trtllm-serve disaggregated_mpi_worker [OPTIONS]

Options

-c, --config, --config_file <config_file>#: beta Path to the disaggregated serving configuration YAML file.

--log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

embeddings#

Run an OpenAI-compatible /v1/embeddings server for encoder-only models.

Coalesces concurrent requests with a dynamic batcher and serves them through the synchronous llm.encode() fast path (no KV cache / sampler / scheduler). Single-GPU only: the command does not expose tensor/pipeline parallelism.

MODEL: model name | HF checkpoint path

Usage

trtllm-serve embeddings [OPTIONS] MODEL

Options

--host <host>#: Hostname of the server.

--port <port>#: Port of the server.

--log_level <log_level>#

The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--max_batch_size <max_batch_size>#: Maximum batch size coalesced into a single encode() call.

--max_num_tokens <max_num_tokens>#: Maximum number of batched input tokens in each encode() call.

--max_queue_delay <max_queue_delay>#: Dynamic-batching hold window in seconds: how long an incoming request waits for others to join its batch before being dispatched (mirrors Triton’s max_queue_delay_microseconds).

--max_queue_size <max_queue_size>#: Maximum number of in-flight queued requests; further requests are rejected with HTTP 429 (mirrors Triton’s max_queue_size).

--trust_remote_code#: Flag for HF transformers.

--config, --extra_llm_api_options <extra_llm_api_options>#: Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file.

--hf_revision, --revision <revision>#: The revision to use for the HuggingFace model (branch name, tag name, or commit id).

--metadata_server_config_file <metadata_server_config_file>#: Path to metadata server config file

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

Arguments

MODEL#: Required argument

mm_embedding_serve#

Running an OpenAI API compatible server

MODEL: model name or Hugging Face checkpoint path

Usage

trtllm-serve mm_embedding_serve [OPTIONS] MODEL

Options

--host <host>#: beta Hostname of the server.

--port <port>#: beta Port of the server.

--log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--max_batch_size <max_batch_size>#: beta Maximum number of requests that the engine can schedule.

--max_num_tokens <max_num_tokens>#: beta Maximum number of batched input tokens after padding is removed in each batch.

--gpus_per_node <gpus_per_node>#: beta Number of GPUs per node. Default to None, and it will be detected automatically.

--trust_remote_code#: beta Flag for HF transformers.

--config, --extra_encoder_options <extra_encoder_options>#: prototype Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file. Prefer –config over –extra_encoder_options.

--hf_revision, --revision <revision>#: beta The revision to use for the HuggingFace model (branch name, tag name, or commit id).

--free_gpu_memory_fraction <free_gpu_memory_fraction>#: beta Free GPU memory fraction reserved for KV Cache, after allocating model weights and buffers.

--tensor_parallel_size, --tp_size <tensor_parallel_size>#: beta Tensor parallelism size.

--metadata_server_config_file <metadata_server_config_file>#: prototype Path to metadata server config file

--allow_request_chat_template#: prototype Allow clients to supply per-request chat_template values. Only enable this for trusted clients.

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

Arguments

MODEL#: Required argument

mooncake_master#

Own a Mooncake pool for as long as this command runs.

Start this before the servers that join the pool. They wait for the manifest, so the order within a job is not delicate, but nothing can hold capacity or serve a lookup until this is up.

Usage

trtllm-serve mooncake_master [OPTIONS]

Options

--rpc_port <rpc_port>#

Port the store clients reach the master on.

Default:: 50051

--metrics_port <metrics_port>#

Prometheus port. Pool occupancy and eviction are read from here or from the master’s log.

Default:: 9004

--eviction_ratio <eviction_ratio>#

Fraction of the pool freed per eviction pass.

Default:: 0.05

--pool_file <pool_file>#: File to publish the pool manifest to once the master answers. Servers name it as mooncake_store.pool: file://<path>, which is how they reach a master whose host the scheduler chose. Removed on exit so a stale address is never dialed. One is written to –run_dir regardless.

--protocol <protocol>#

Transport every participant will use: ‘rdma’ or ‘tcp’. Recorded in the manifest so the servers need not restate it. TCP is for bring-up only; it invalidates performance conclusions.

Default:: 'rdma'

--metadata_server <metadata_server>#

Mooncake metadata service, recorded in the manifest. P2PHANDSHAKE keeps a separate metadata process out of the deployment.

Default:: 'P2PHANDSHAKE'

--namespace <namespace>#

Default key namespace for this pool, recorded in the manifest. A server may override it.

Default:: 'trtllm'

--run_dir <run_dir>#: Where to keep the master’s log and a copy of the manifest. Defaults to a temporary directory.

--binary <binary>#

The master executable to run, looked up on PATH. It ships with the Mooncake runtime that docker/common/install_mooncake.sh installs.

Default:: 'mooncake_master'

--timeout <timeout>#

Seconds to wait for the master to accept connections before giving up on it.

Default:: 60.0

--heartbeat_seconds <heartbeat_seconds>#

Interval between liveness lines. 0 disables them.

Default:: 300

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

mooncake_pool_report#

Report the capacity a run’s pool actually had, and where blocks landed.

Safe to run after the job, since the records outlive the processes that wrote them.

Usage

trtllm-serve mooncake_pool_report [OPTIONS]

Options

--run_dir <run_dir>#: Required Directory holding the pool manifest and the segments/ records each participating rank wrote.

--master_log <master_log>#: The master’s log, for the block-placement table. Defaults to mooncake_master.log in –run_dir.

--telemetry, --no-telemetry#: Enable or disable anonymous usage telemetry collection.

serve#

Running an OpenAI API compatible server

MODEL: model name or Hugging Face checkpoint path

Usage

trtllm-serve serve [OPTIONS] MODEL

Options

--tokenizer <tokenizer>#: beta Path or name of the tokenizer. When using the PyTorch backend, this replaces the default HuggingFace tokenizer.

--custom_tokenizer <custom_tokenizer>#: prototype Custom tokenizer type: alias (e.g., ‘deepseek_v32’) or Python import path (e.g., ‘tensorrt_llm.tokenizer.deepseek_v32.DeepseekV32Tokenizer’).

--post_processor_hook <post_processor_hook>#: prototype Python import path of a user post-processing hook applied after detokenization and before the per-endpoint response formatter (e.g. ‘my_pkg.guardrail.MyPostProcessorHook’). The class must be importable and picklable, take no constructor arguments, and be callable as ‘__call__(chunk) -> verdict’ (see tensorrt_llm.executor.postprocessor_hook). It runs once per output, per streaming chunk, and may rewrite, suppress, or terminate the output; it owns its own per-request state.

--host <host>#: beta Hostname of the server.

--port <port>#: beta Port of the server.

--backend <backend>#

deprecated The backend to use to serve the model. Default is pytorch backend.

Options:: pytorch

--generation-config <generation_config>#

prototype Sampling defaults source. ‘auto’ loads supported values from the model’s generation_config.json; ‘trtllm’ uses TRT-LLM defaults.

Options:: auto | trtllm

--custom_module_dirs <custom_module_dirs>#: prototype Paths to custom module directories to import.

--log_level <log_level>#

beta The logging level.

Options:: internal_error | error | warning | info | verbose | debug | trace

--max_beam_width <max_beam_width>#: beta Maximum number of beams for beam search decoding.

--max_batch_size <max_batch_size>#: beta Maximum number of requests that the engine can schedule.

--max_num_tokens <max_num_tokens>#: beta Maximum number of batched input tokens after padding is removed in each batch.

--max_seq_len <max_seq_len>#: beta Maximum total length of one request, including prompt and outputs. If unspecified, the value is deduced from the model config.

--tensor_parallel_size, --tp_size <tensor_parallel_size>#: beta Tensor parallelism size.

--pipeline_parallel_size, --pp_size <pipeline_parallel_size>#: beta Pipeline parallelism size.

--context_parallel_size, --cp_size <context_parallel_size>#: beta Context parallelism size.

--moe_expert_parallel_size, --ep_size <moe_expert_parallel_size>#: beta expert parallelism size

--moe_cluster_parallel_size, --cluster_size <moe_cluster_parallel_size>#: deprecated [Deprecated] Expert cluster parallelism size. This option is no longer supported and will be removed in a future release.

--gpus_per_node <gpus_per_node>#: beta Number of GPUs per node. Default to None, and it will be detected automatically.

--free_gpu_memory_fraction, --kv_cache_free_gpu_memory_fraction <free_gpu_memory_fraction>#: beta Free GPU memory fraction reserved for KV Cache, after allocating model weights and buffers.

--kv_cache_dtype <kv_cache_dtype>#

prototype KV cache quantization dtype for PyTorch backend. ‘auto’ uses checkpoint/model metadata; explicit values force override.

Options:: auto | fp8 | nvfp4

--num_postprocess_workers <num_postprocess_workers>#: prototype Number of workers to postprocess raw responses to comply with OpenAI protocol.

--num_serve_frontends <num_serve_frontends>#: prototype Number of HTTP or OpenEngine frontend processes serving one executor; values > 1 share the serving port via SO_REUSEPORT (classic IPC executor path only).

--num_input_processor_workers <num_input_processor_workers>#: prototype Size of the dedicated thread pool that runs the HF input processor (multimodal preprocess) on the chat and completion endpoints.

--num_media_load_workers <num_media_load_workers>#: prototype Size of the dedicated thread pool that decodes media payloads (image / video / audio) for multimodal requests.

--trust_remote_code#: beta Flag for HF transformers.

--hf_revision, --revision <revision>#: beta The revision to use for the HuggingFace model (branch name, tag name, or commit id). Prefer –hf_revision over –revision.

--config, --extra_llm_api_options <extra_llm_api_options>#: prototype Path to a YAML configuration file. Explicit CLI flags take precedence over values in this file, while –set overrides both. Can be specified as either –config or –extra_llm_api_options.

--set <config_overrides>#: prototype Override a canonical LlmArgs path as PATH=YAML_VALUE. Repeatable; applied after –config and dedicated CLI flags. Not supported for startup, telemetry, environment, server-only, or VisualGen settings.

--reasoning_parser <reasoning_parser>#

prototype Specify the parser for reasoning models. Use ‘auto’ to automatically select based on the model.

Options:: auto | minimax_m2_append_think | minimax_m2 | qwen3_5 | qwen3 | deepseek-r1 | glm_moe_dsa | glm47 | glm45 | glm | deepseek_v4 | deepseek_v41 | poolside_v1 | laguna | minimax_m3 | k-exaone | nano-v3 | nemotron-v3 | gemma4 | kimi_k25 | kimi_k2 | kimi_k3

--tool_parser <tool_parser>#

prototype Specify the parser for tool models. Use ‘auto’ to automatically select based on the model.

Options:: auto | qwen3 | qwen3_coder | kimi_k2 | kimi_k3 | deepseek_v3 | deepseek_v31 | deepseek_v32 | deepseek_v4 | deepseek_v41 | gemma4 | glm4 | glm47 | minimax_m2 | minimax_m3 | poolside_v1

--metadata_server_config_file <metadata_server_config_file>#: prototype Path to metadata server config file

--server_role <server_role>#: prototype Server role for disaggregated serving. CONTEXT=prefill (prompt processing), GENERATION=decode (token generation), MM_ENCODER=multimodal encoder, VISUAL_GEN=visual generation. Required when using service registry.

--otlp_traces_endpoint <otlp_traces_endpoint>#: prototype Target URL to which OpenTelemetry traces will be sent.

--telemetry, --no-telemetry#: beta Enable or disable anonymous usage telemetry collection.

--disagg_cluster_uri <disagg_cluster_uri>#: prototype URI of the disaggregated cluster.

--enable_chunked_prefill#: prototype Enable chunked prefill

--enable_attention_dp#: beta Enable attention data parallel.

--enable_locality_domains#: prototype Enable locality domain execution for supported PyTorch backend ops.

--media_io_kwargs <media_io_kwargs>#: prototype Keyword arguments for media I/O as a JSON string. Keys are modality names (“video”, “image”, “audio”) whose values are dicts of keyword arguments forwarded to the corresponding loader. Example: ‘{“video”: {“extract_audio”: true, “num_frames”: 16}}’ to enable audio extraction from video files.

--video_pruning_rate <video_pruning_rate>#: prototype Pruning rate for video frames in multimodal models. Applied by Efficient Video Sampling (EVS). None disables EVS, values in [0, 1) enable pruning.

--chat_template <chat_template>#: prototype Specify a custom chat template. Can be a file path or one-liner template string

--allow_request_chat_template#: prototype Allow clients to supply per-request chat_template values. Only enable this for trusted clients.

--middleware <middleware>#: prototype FastAPI middleware import path to add to the server app. Can be specified multiple times. Each value must point to either a middleware class or an async HTTP middleware function.

--grpc#: prototype Run the selected gRPC protocol instead of the OpenAI HTTP server.

--grpc-protocol <grpc_protocol>#

prototype Protocol used when –grpc is enabled.

Options:: smg | openengine

--openengine-enable-load-metrics#: prototype Expose per-rank KV occupancy for OpenEngine routing, independently of KV events.

--served_model_name <served_model_name>#: prototype The model name used in the API. If not specified, the model path is used as the model name. This is useful when the model path is long or when you want to expose a custom name to clients.

--enable_visual_gen#: prototype Enable VisualGen runtime for model checkpoints that support both LLM and Visual Generation. Not required if –visual_gen_args specified or the model supports Visual Generation only.

--visual_gen_args <visual_gen_args>#: prototype Path to a YAML file with VisualGen engine args.

--agent_percentage <agent_percentage>#: prototype The percentage of agent requests to schedule. Defaults to 0.0. Should be between 0.0 and 1.0.

--agent_types <agent_types>#: prototype Types of agents to schedule. Now Only Support Open Deep Research agent.

--report_addr <report_addr>#: prototype Write the host:port this server actually bound to this file, atomically, once the socket is bound. Lets –port 0 be used and have the launcher read the kernel-assigned port back instead of reserving one up front.

--enable_sleep_mode#: prototype Enable runtime-memory checkpointing and authenticated HTTP control endpoints.

Arguments

MODEL#: Required argument

Besides the above examples, trtllm-serve is also used as an entrypoint for performance benchmarking. Please refer to Performance Benchmarking with `trtllm-serve <NVIDIA/TensorRT-LLM>` for more details.

trtllm-serve#

About#

Starting a Server#

Inference Endpoints#

Chat API#

Completions API#

Responses API#

Multimodal Serving#

Multimodal Chat API#

Multimodal Modality Coverage#

Visual Generation Serving#

Multi-node Serving with Slurm#

Metrics Endpoint#

Runtime Profiling Endpoints#

POST /start_profile#

POST /stop_profile#

Example#

Configuring with YAML Files and --set#

Editor Validation of YAML Files#

Syntax#

trtllm-serve#

disaggregated#

disaggregated_mpi_worker#

embeddings#

mm_embedding_serve#

mooncake_master#

mooncake_pool_report#

serve#

Configuring with YAML Files and `--set`#