Plugin Utils#
-
template<typename T, class Enable = void>
struct Serializer# Generic serializer template for plugin data.
Provides serialization/deserialization for plugin state. Specialized for arithmetic and enum types.
- Template Parameters:
T – Type to serialize
Enable – SFINAE enabler
-
template<typename T>
struct Serializer<T, typename std::enable_if_t<std::is_arithmetic_v<T> || std::is_enum_v<T>>># Serializer specialization for arithmetic and enum types.
-
void trt_edgellm::plugins::applyThorSMRenumberWAR(int32_t &smVersion)#
Workaround for CUDA12/13 Thor SM re-numbering.
- Parameters:
smVersion – CUDA SM version that may be normalized in-place.
- PluginField trt_edgellm::plugins::layerNameField(
- std::string const &layerName
The kLAYER_NAME_FIELD entry of getFieldsToSerialize(); it points into
layerName.
- std::string trt_edgellm::plugins::describePluginLayer(
- char const *kind,
- std::string const &layerName,
- std::string const &shape
A layer in plugin error logs: “<kind> ‘<name>’ (<shape>)”, or “<kind> (<shape>)” when
layerNameis empty.
- void trt_edgellm::plugins::reportCompactWorkspace(
- char const *kind,
- std::string const &layerName,
- bool reserved
Record in the open compact build report whether the phase-8 layer
layerNameof pluginkindreserved the compact workspace; a no-op when no report is open. Called from getWorkspaceSize(), which TensorRT may call more than once per layer.
-
size_t trt_edgellm::plugins::alignTensorSize(size_t size)#
Align a byte size to the device workspace alignment.
- size_t trt_edgellm::plugins::accumulateWorkspaceSize(
- size_t currentSize,
- rt::Coords const &shape,
- nvinfer1::DataType dataType
Accumulate workspace size for a given shape and data type. Device alignment will be applied automatically.
- Parameters:
currentSize – Current workspace size
shape – Tensor shape
dataType – Tensor data type
- Returns:
Accumulated workspace size that aligned to device alignment.
- rt::Tensor trt_edgellm::plugins::assignTensorFromWorkspace(
- std::byte *&workspace,
- rt::Coords const &shape,
- nvinfer1::DataType dataType
Given a contiguous workspace, assign (non-owned) tensor with specified shape and data type from the start of workspace. After assignment, the workspace pointer will shift the size of tensor and align to device alignment.
- Parameters:
workspace – The contiguous workspace pointer
shape – Requested Tensor shape
dataType – Requested Tensor data type
- Returns:
Assigned tensor
Warning
doxygenfunction: Unable to resolve function “trt_edgellm::plugins::resolvePluginLayerName” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:
- std::string resolvePluginLayerName(char const *name, PluginFieldCollection const *fc)
- std::string resolvePluginLayerName(char const *name, nvinfer1::PluginFieldCollection const *fc)
-
template<typename T>
nvinfer1::PluginFieldType trt_edgellm::plugins::toFieldType( Convert C++ type to TensorRT PluginFieldType.
- Template Parameters:
T – C++ type
- Returns:
Corresponding PluginFieldType