Plugin Utils#

template<typename T, class Enable = void>
struct Serializer#

Generic serializer template for plugin data.

Provides serialization/deserialization for plugin state. Specialized for arithmetic and enum types.

Template Parameters:
  • T – Type to serialize

  • Enable – SFINAE enabler

template<typename T>
struct Serializer<T, typename std::enable_if_t<std::is_arithmetic_v<T> || std::is_enum_v<T>>>#

Serializer specialization for arithmetic and enum types.

Public Static Functions

static inline void serialize(std::byte **buffer, T const &value)#

Serialize value to buffer.

Parameters:
  • buffer – Output buffer pointer (advanced after write)

  • value – Value to serialize

static inline void deserialize(
std::byte const **buffer,
size_t *buffer_size,
T *value
)#

Deserialize value from buffer.

Parameters:
  • buffer – Input buffer pointer (advanced after read)

  • buffer_size – Buffer size (decremented after read)

  • value – Output value

void trt_edgellm::plugins::applyThorSMRenumberWAR(int32_t &smVersion)#

Workaround for CUDA12/13 Thor SM re-numbering.

Parameters:

smVersion – CUDA SM version that may be normalized in-place.

PluginField trt_edgellm::plugins::layerNameField(
std::string const &layerName
)#

The kLAYER_NAME_FIELD entry of getFieldsToSerialize(); it points into layerName.

std::string trt_edgellm::plugins::describePluginLayer(
char const *kind,
std::string const &layerName,
std::string const &shape
)#

A layer in plugin error logs: “<kind> ‘<name>’ (<shape>)”, or “<kind> (<shape>)” when layerName is empty.

void trt_edgellm::plugins::reportCompactWorkspace(
char const *kind,
std::string const &layerName,
bool reserved
) noexcept#

Record in the open compact build report whether the phase-8 layer layerName of plugin kind reserved the compact workspace; a no-op when no report is open. Called from getWorkspaceSize(), which TensorRT may call more than once per layer.

size_t trt_edgellm::plugins::alignTensorSize(size_t size)#

Align a byte size to the device workspace alignment.

size_t trt_edgellm::plugins::accumulateWorkspaceSize(
size_t currentSize,
rt::Coords const &shape,
nvinfer1::DataType dataType
)#

Accumulate workspace size for a given shape and data type. Device alignment will be applied automatically.

Parameters:
  • currentSize – Current workspace size

  • shape – Tensor shape

  • dataType – Tensor data type

Returns:

Accumulated workspace size that aligned to device alignment.

rt::Tensor trt_edgellm::plugins::assignTensorFromWorkspace(
std::byte *&workspace,
rt::Coords const &shape,
nvinfer1::DataType dataType
)#

Given a contiguous workspace, assign (non-owned) tensor with specified shape and data type from the start of workspace. After assignment, the workspace pointer will shift the size of tensor and align to device alignment.

Parameters:
  • workspace – The contiguous workspace pointer

  • shape – Requested Tensor shape

  • dataType – Requested Tensor data type

Returns:

Assigned tensor

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::plugins::resolvePluginLayerName” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches:

- std::string resolvePluginLayerName(char const *name, PluginFieldCollection const *fc)
- std::string resolvePluginLayerName(char const *name, nvinfer1::PluginFieldCollection const *fc)
template<typename T>
nvinfer1::PluginFieldType trt_edgellm::plugins::toFieldType(
)#

Convert C++ type to TensorRT PluginFieldType.

Template Parameters:

T – C++ type

Returns:

Corresponding PluginFieldType