External Weight Manager#
-
class ExternalWeightManager#
Owns immutable, final-layout model weights prepared during initialization.
The checkpoint path allocates one persistent arena, then writes every final engine input directly from bounded CUDA-mapped checkpoint ranges. Each range is released after its transform; once load() returns, later methods only validate or publish stable arena addresses.
Public Functions
-
ExternalWeightManager() = default#
-
ExternalWeightManager(ExternalWeightManager const&) = delete#
- ExternalWeightManager &operator=(
- ExternalWeightManager const&
-
ExternalWeightManager(ExternalWeightManager&&) noexcept = default#
- ExternalWeightManager &operator=( ) noexcept = default#
- void load(
- std::filesystem::path const &engineDir,
- std::filesystem::path const &configPath,
- cudaStream_t stream,
- std::filesystem::path const &componentCheckpointDir = {},
- std::filesystem::path const &targetCheckpointDir = {}
Load plugin-ready sidecars or load and transform an original checkpoint.
componentCheckpointDir overrides the checkpoint recorded while building this engine. targetCheckpointDir supplies target-owned fallback weights for a separately packaged draft.
The stream is synchronized before this method returns, leaving only engine-input tensors in their final plugin layouts.
- void validateAgainstEngine(
- EngineExecutor const &executor,
- std::string_view engineLabel
- void bindToContext(
- nvinfer1::ICudaEngine const &engine,
- nvinfer1::IExecutionContext &context,
- std::string_view engineLabel
Validate against a raw engine and point its inputs at the loaded weights.
Encoder runners drive TensorRT directly instead of through EngineExecutor and TensorMap, so they bind here. Weight addresses are immutable for the life of the context, so one call at load time is enough.
-
std::optional<Tensor> takeEmbedding()#
Transfer the checkpoint-backed embedding prepared during load(), if any.
-
inline size_t size() const noexcept#
-
ExternalWeightManager() = default#