RequestOutput

class tensorrt_llm.llmapi.RequestOutput

Bases: DetokenizedGenerationResultBase, GenerationResult

The output data of a completion request to the LLM.

request_id

The unique ID of the request.

Type:

int

prompt

The prompt string of the request.

Type:

str, optional

prompt_token_ids

The token ids of the prompt.

Type:

List[int]

outputs

The output sequences of the request.

Type:

List[CompletionOutput]

context_logits

The logits on the prompt token ids.

Type:

torch.Tensor, optional

disaggregated_params

Parameters for disaggregated serving, including multimodal embedding handles.

Type:

DisaggregatedParams, optional

finished

Whether the whole request is finished.

Type:

bool

error

The error message if this result completed with an error.

Type:

str, optional

class PostprocWorker(

pull_pipe_addr: tuple[str, bytes | None],

push_pipe_addrs: List[tuple[str, bytes | None]],

tokenizer_dir: str,

record_creator: Callable[[Input, TransformersTokenizer], Any],

post_processor_hook: str | None = None,

)

Bases: object

The worker to postprocess the responses from the executor’s await_response.

class Input(

rsp: ForwardRef('tllm.Response') | ForwardRef('ResponseWrapper'),

sampling_params: tensorrt_llm.sampling_params.SamplingParams | None = None,

postproc_params: tensorrt_llm.executor.postproc_worker.PostprocParams | None = None,

disaggregated_params: ForwardRef('DisaggregatedParams') | None = None,

streaming: bool | None = None,

)

Bases: object

__init__(

rsp: tllm.Response | ResponseWrapper,

sampling_params: SamplingParams | None = None,

postproc_params: PostprocParams | None = None,

disaggregated_params: DisaggregatedParams | None = None,

streaming: bool | None = None,

) → None

disaggregated_params: DisaggregatedParams | None = None

postproc_params: PostprocParams | None = None

rsp: tllm.Response | ResponseWrapper

sampling_params: SamplingParams | None = None

streaming: bool | None = None

class Output(

client_id,

res,

is_final,

metrics,

request_perf_metrics,

disaggregated_params,

should_abort,

finish_reason,

num_generated_tokens,

)

Bases: NamedTuple

count(value, /)

Return number of occurrences of value.

index(

value,

start=0,

stop=9223372036854775807,

/,

)

Return first index of value.

Raises ValueError if the value is not present.

client_id: int

Alias for field number 0

disaggregated_params: Any

Alias for field number 5

finish_reason: str | None

Alias for field number 7

is_final: bool

Alias for field number 2

metrics: dict[str, float] | None

Alias for field number 3

num_generated_tokens: int | None

Alias for field number 8

request_perf_metrics: Any

Alias for field number 4

res: Any

Alias for field number 1

should_abort: bool

Alias for field number 6

__init__(

pull_pipe_addr: tuple[str, bytes | None],

push_pipe_addrs: List[tuple[str, bytes | None]],

tokenizer_dir: str,

record_creator: Callable[[Input, TransformersTokenizer], Any],

post_processor_hook: str | None = None,

)

Parameters:

• pull_pipe_addr (tuple[str, Optional[bytes]]) – The address and HMAC key of the input IPC.

• push_pipe_addrs – The addresses and HMAC keys of the output IPC lanes, one per frontend (a single-element list in single-frontend mode).

• tokenizer_dir (str) – The directory to load tokenizer.

• record_creator (Callable[["ResponsePostprocessWorker.Input"], Any]) – A creator for creating a record for a request.

• result_handler (Optional[Callable[[GenerationResultBase], Any]]) – A callback handles the final result.

• post_processor_hook (Optional[str]) – Import path of the user post-processing hook; built once and threaded onto each record.

static default_record_creator(

inp: PostprocWorker.Input,

tokenizer: TransformersTokenizer,

) → DetokenizedGenerationResultBase

start()

Start the workflow in the current thread.

__init__() → None

abort() → None

Abort the generation request.

aborted() → bool

Return whether the generation request is aborted.

Returns:

whether the generation request is aborted.

Return type:

bool

async aresult() → GenerationResult

Wait for the completion of the request, and return the result.

Returns:

generation result.

Return type:

tensorrt_llm.executor.result.GenerationResult

clear_logprob_params() → None

do_tracing(

output: CompletionOutput,

req_perf_metrics_dict: dict[str, float] | None = None,

) → None

Perform distributed tracing for the generation request.

Parameters:

• output (CompletionOutput) – The output of the generation result.

• req_perf_metrics_dict (Optional[dict[str, float]]) – Request performance metrics. Defaults to None.

record_stats(

output: CompletionOutput,

stats: dict[str, float] | None = None,

sequence_index: int = 0,

) → None

Record the stats of the generation result.

Called once per candidate when it finishes. When n > 1 each candidate has its own timestamps so TPOT and GENERATION_TOKENS are computed independently per candidate. PROMPT_TOKENS are only recorded for sequence_index == 0 to avoid double-counting the shared prompt.

Parameters:

• output (CompletionOutput) – The output of the generation result.

• stats (Optional[dict[str, float]]) – The stats of the generation result. Defaults to None.

• sequence_index (int) – Index of this candidate (0 for the first / only sequence). Defaults to 0.

result(

timeout: float | None = None,

) → GenerationResult

Wait for the completion of the request, and return the result.

Parameters:

timeout (float, optional) – The maximum number of seconds to wait for the request to complete. None (default) waits indefinitely. The timeout is a total budget across all streaming steps, not per-step.

Returns:

generation result.

Return type:

tensorrt_llm.executor.result.GenerationResult

Raises:

TimeoutError – If the request does not complete within timeout seconds. Bounding the wait prevents a silently-dead executor worker from hanging the caller forever.

property context_logits: Tensor | None

property disaggregated_params: DisaggregatedParams | None

Returns the disaggregated params.

property error: str | None

Return the error message if this result completed with an error.

property finished: bool

property outputs: List[CompletionOutput]

property prompt: str | None

property prompt_token_ids: List[int]

property request_id: int