LLM Builder

class LLMBuilder

Builder class for Large Language Model TensorRT engines. Handles the complete process of building TensorRT engines from ONNX models for various types of LLMs including standard, speculative-decoding, and VLM models.

Public Functions

LLMBuilder(

std::filesystem::path const &onnxDir,

std::filesystem::path const &engineDir,

LLMBuilderConfig const &config

)

Constructor for LLMBuilder.

Parameters:

• onnxDir – Directory containing the ONNX model and configuration files

• engineDir – Directory where the built engine and related files will be saved

• config – Configuration object specifying build parameters

~LLMBuilder() noexcept = default

Destructor.

bool build()

Build the TensorRT engine from the ONNX model. This method performs the complete build process including:

• Loading and parsing the ONNX model

• Setting up optimization profiles

• Building the TensorRT engine

• Copying necessary files to the engine directory

Throws:

std::filesystem::filesystem_error – If filesystem operations fail

Returns:

true if build was successful, false otherwise

struct RaggedProfilePoint

Public Members

int64_t numSequences

int64_t physicalTokens

int64_t queryOffsets

int64_t logitsRows

struct RaggedProfileRange

Public Members

RaggedProfilePoint min

RaggedProfilePoint opt

RaggedProfilePoint max

struct LLMBuilderConfig

Configuration structure for LLM model building. Contains all parameters needed to configure the TensorRT engine building process for Large Language Models, including standard and speculative-decoding engines.

Public Functions

inline int64_t resolvedRoleQueryLength() const

inline int64_t resolvedMaxQueryLength() const

inline int64_t checkedPhysicalTokens(int64_t queryLength) const

inline int64_t resolvedMaxPhysicalTokens() const

inline RaggedProfileRange raggedPrefillProfileRange() const

inline RaggedProfileRange raggedDecodeProfileRange() const

inline RaggedProfileRange raggedMultiTokenGenerationProfileRange(

int64_t maxTokensPerSequence

) const

inline int64_t resolvedKVPoolPages() const

Resolve the exact physical K-page count serialized into the engine binding shape.

Throws:

std::runtime_error – for invalid dimensions, overflow, or a below-minimum override

Returns:

maxKVPoolPages, or the minimum active pages when it is zero

inline int32_t resolveNumSwaPages(int32_t slidingWindowCapacity)

Resolve and store the positive SWA page budget once SWA capability markers are known. A zero configured value selects the bounded builder default; runtime config does not use this auto mode.

inline std::array<int64_t, 3> resolveKVPoolPageProfile(

int32_t kvCacheCapacity

) const

Resolve the min/opt/max page dimension for one KV-cache input profile. SWA-capable layers optimize for the smaller storage policy while the profile covers both page counts.

inline Json toJson() const

Convert configuration to JSON format for serialization.

Returns:

JSON object containing all configuration parameters

inline std::string toString() const

Convert configuration to human-readable string format.

Returns:

String representation of the configuration for debugging/logging

Public Members

int64_t maxInputLen = {1024}

Maximum input sequence length for the model.

bool specDraft = {false}

Whether this is a speculative-decoding draft model.

bool specBase = {false}

Whether this is a speculative-decoding base model.

int64_t maxBatchSize = {4}

Maximum batch size for inference.

int64_t maxLoraRank = {0}

Maximum LoRA rank (0 = no LoRA support)

int64_t maxKVCacheCapacity = {4096}

Maximum KV cache capacity (sequence length)

int64_t maxVerifyTreeSize = {60}

Maximum length of input_ids passed into spec base model for verification.

int64_t maxDraftTreeSize = {60}

Maximum length of input_ids passed into spec draft model for draft generation.

bool profilingDetailed = {false}

Enable detailed profiling verbosity for layer info extraction

int64_t maxKVPoolPages = {0}

Exact physical K-page count for the engine’s KV pool. Zero selects the minimum active pages. Kept after the original aggregate fields so existing positional initializers remain source-compatible.

std::string raggedBackend = {"entry_padded_compatibility"}

int64_t maxQueryLength = {0}

Additional model-role query width (for example, a diffusion canvas). Zero selects role-derived sizing.

int64_t tpSize = {1}

Tensor parallel size.

int64_t tpRank = {0}

Tensor parallel rank

int64_t numSwaPages = {0}

Total K-page budget for the independent SWA pool (0 = auto-size when reduced markers are present).

Public Static Functions

static inline LLMBuilderConfig fromJson(Json const &json)

Create configuration from JSON format.

Parameters:

json – JSON object containing configuration parameters

Throws:

nlohmann::json::type_error – If JSON value types don’t match expected types

Returns:

LLMBuilderConfig object with parsed parameters