External Weight Manager#

class ExternalWeightManager#

Owns immutable, final-layout model weights prepared during initialization.

Checkpoint transforms write directly from bounded CUDA-mapped source ranges. Inputs, embeddings and plugin resources use a persistent arena; refit weights use a separate temporary arena transferred to engine initialization.

Public Functions

ExternalWeightManager() = default#
~ExternalWeightManager() noexcept#
ExternalWeightManager(ExternalWeightManager const&) = delete#
ExternalWeightManager &operator=(
ExternalWeightManager const&
) = delete#
ExternalWeightManager(ExternalWeightManager &&other) noexcept#
ExternalWeightManager &operator=(
ExternalWeightManager &&other
) noexcept#
void load(
std::filesystem::path const &engineDir,
std::filesystem::path const &configPath,
cudaStream_t stream,
std::filesystem::path const &componentCheckpointDir = {},
std::filesystem::path const &targetCheckpointDir = {},
std::optional<int32_t> tpRank = std::nullopt,
std::optional<int32_t> tpSize = std::nullopt,
std::function<void()> const &checkCancelled = {}
)#

Load plugin-ready sidecars or load and transform an original checkpoint.

componentCheckpointDir overrides the checkpoint recorded while building this engine. targetCheckpointDir supplies target-owned fallback weights for a separately packaged draft.

The stream is synchronized before returning. All tensors have final layouts; temporary refit storage remains owned here until takeRefitWeights(). checkCancelled runs synchronously between completed weight operations and may throw to abort loading.

void validateAgainstEngine(
EngineExecutor const &executor,
std::string_view engineLabel
)#
void registerTensorMapEntries(TensorMap &map)#
void bindToContext(
nvinfer1::ICudaEngine const &engine,
nvinfer1::IExecutionContext &context,
std::string_view engineLabel
)#

Validate against a raw engine and point its inputs at the loaded weights.

Encoder runners drive TensorRT directly instead of through EngineExecutor and TensorMap, so they bind here. Weight addresses are immutable for the life of the context, so one call at load time is enough.

std::optional<Tensor> takeEmbedding()#

Transfer the checkpoint-backed embedding prepared during load(), if any.

std::unique_ptr<RefitWeightSet> takeRefitWeights()#

Transfer temporary refit storage once after load; null for ordinary engines. The returned owner must survive refit completion. Throws before load or on repeated transfer.

inline size_t size() const noexcept#