Model Artifacts

struct ModelArtifacts

Everything LLMInferenceRuntime reads off disk before it can assemble itself: the parsed deployment configuration, the engines, and the weight-shaped files that sit next to them.

Loading is separated from assembly so a runtime can also be built from artifacts that no engine directory produced — a substitute EngineExecutor plus a hand-written DeploymentConfig is a complete input.

Public Members

DeploymentConfig deployment

std::unique_ptr<EngineExecutor> baseExecutor

std::unique_ptr<EngineExecutor> draftExecutor

Null unless the deployment drafts speculatively.

ExternalWeightManager weights

ExternalWeightManager draftWeights

Externalized draft-engine weights, already validated against draftExecutor. Empty for vanilla deployments, and also empty when the draft was exported without --externalize-weights. The selected speculative decoder publishes it into its own tensor map.

EmbeddingData embedding

std::optional<Tensor> pleEmbedding

Gemma4 per-layer embedding; absent unless the base engine enables PLE.

std::optional<Tensor> vocabMap

Reduced-vocab mapping table; absent unless the base vocab is reduced.

std::unique_ptr<tokenizer::Tokenizer> tokenizer

std::filesystem::path checkpointDir

Checkpoint directories that supplied weights. Retained because encoder runners load from them separately.

std::filesystem::path draftCheckpointDir

Public Static Functions

static ModelArtifacts loadFromEngineDir(

std::filesystem::path const &engineDir,

std::optional<SpecDecodeDraftingConfig> const &draftingConfig,

std::filesystem::path const &checkpointDir,

std::filesystem::path const &draftCheckpointDir,

cudaStream_t stream

)

Read one engine directory and validate each loaded engine against its parsed configuration.

draftCheckpointDir must be empty for native MTP, whose draft weights are part of checkpointDir; the returned artifacts record it as equal to checkpointDir in that case.