Tensor#

class Coords#

Array of dimensions that used to store the shape of a tensor. Support up to 8 dimensions with all dimensions are non-negative. Default constructor create empty coords with 0 dimensions and 0 volumes.

Public Functions

Coords() = default#

Default constructor.

inline Coords(Coords const &coords) noexcept#

Copy constructor.

Parameters:

coords – Coordinates to copy

Coords(Coords &&coords) noexcept = default#

Move constructor.

Coords &operator=(Coords const &coords) noexcept = default#

Copy assignment operator.

Parameters:

coords – Coordinates to copy

Returns:

Reference to this

Coords &operator=(Coords &&coords) noexcept = default#

Move assignment operator.

Parameters:

coords – Coordinates to move

Returns:

Reference to this

inline Coords(nvinfer1::Dims const &dims)#

Construct from TensorRT Dims.

Parameters:

dims – TensorRT dimensions

template<typename IT>
inline Coords(IT begin, IT end)#

Construct from iterator range.

Template Parameters:

IT – Iterator type

Parameters:

  • begin – Beginning of range

  • end – End of range

Throws:

std::runtime_error – if number of dimensions exceeds kMAX_DIMS

inline Coords(std::initializer_list<int64_t> init)#

Construct from initializer list.

Parameters:

init – Initializer list of dimensions

inline Coords(std::vector<int64_t> const &vec)#

Construct from vector.

Parameters:

vec – Vector of dimensions

inline int32_t getNumDims() const noexcept#

Get number of dimensions.

Returns:

Number of dimensions

inline int64_t &operator[](int32_t idx)#

Array subscript operator (mutable)

Parameters:

idx – Index to access

Throws:

std::out_of_range – if index is out of bounds

Returns:

Reference to dimension at index

inline int64_t operator[](int32_t idx) const#

Array subscript operator (const)

Parameters:

idx – Index to access

Throws:

std::out_of_range – if index is out of bounds

Returns:

Dimension value at index

int64_t volume() const#

Calculate total volume (product of all dimensions)

Returns:

Volume of the coordinates

nvinfer1::Dims getTRTDims() const#

Convert to TensorRT Dims.

Returns:

TensorRT dimensions

std::string formatString() const#

Format coordinates as string.

Returns:

String representation of coordinates

class Tensor#

Tensor class that wrap linear layout tensor. The underlying memory can either be owned by the tensor object or be reused from another allocation. The Tensor Object support reshapes when memory is owned by the object and has sufficient capacity. The default constructor creates an empty tensor object with zero volume with no underlying memory.

Public Functions

Tensor() = default#

Default constructor.

Tensor(Tensor const &other) = delete#

Disable copy constructor to enforce explicit memory ownership transfer.

Non-owned tensor object can be constructed explicitly to workaround the limitation of deleted copy constructor.

Tensor &operator=(Tensor const &other) = delete#

Disable copy assignment operator to enforce explicit memory ownership transfer.

Returns:

Reference to this

Tensor(Tensor &&other) noexcept#

Move constructor allows transfer of memory ownership.

Parameters:

other – Tensor to move from

Tensor &operator=(Tensor &&other) noexcept#

Move assignment operator allows transfer of memory ownership.

Parameters:

other – Tensor to move from

Returns:

Reference to this

~Tensor()#

Destructor.

Tensor(
Coords const &extent,
DeviceType deviceType,
nvinfer1::DataType dataType,
std::string const &name = ""
)#

Constructor that allocates memory on the specified device.

The memory is owned by the tensor object and will be freed when the tensor object is destroyed.

Parameters:

  • extent – The shape of the tensor (must have non-zero volume)

  • deviceType – The device type to allocate memory on

  • dataType – The data type of the tensor (sub-types like kInt4 or kE2M1 are not supported)

  • name – Optional name for the tensor

Tensor(
void *data,
Coords const &extent,
DeviceType deviceType,
nvinfer1::DataType dataType,
std::string const &name = ""
) noexcept#

Constructor that reuses external memory.

Memory is not owned by the tensor object. The caller must ensure the lifecycle of the memory.

Parameters:

  • data – Pointer to existing memory

  • extent – The shape of the tensor

  • deviceType – The device type of the memory

  • dataType – The data type of the tensor

  • name – Optional name for the tensor

Coords getShape() const noexcept#

Get the shape of the tensor.

Returns:

Coordinates representing the tensor shape

DeviceType getDeviceType() const noexcept#

Get the device type.

Returns:

Device type (CPU or GPU)

nvinfer1::DataType getDataType() const noexcept#

Get the data type.

Returns:

TensorRT data type

nvinfer1::Dims getTRTDims() const noexcept#

Get TensorRT dimensions.

Returns:

TensorRT Dims object

bool getOwnMemory() const noexcept#

Check if tensor owns its memory.

Returns:

True if memory is owned, false otherwise

bool isEmpty() const noexcept#

Check if tensor is empty.

Returns:

True if tensor is empty, false otherwise

std::string const &getName() const noexcept#

Get the name of the tensor.

Returns:

Tensor name

int64_t getMemoryCapacity() const noexcept#

Get memory capacity of the underlying buffer.

Returns the memory capacity when the instance was constructed. The value can differ from getShape().volume() * sizeof(dataType) when the tensor is reshaped.

Returns:

Memory capacity in bytes

int64_t getStride(int32_t idx) const#

Get stride of the tensor at given dimension.

Parameters:

idx – Dimension index

Returns:

Stride value

void const *rawPointer() const noexcept#

Get raw data pointer (const)

Returns:

Const pointer to tensor data

void *rawPointer() noexcept#

Get raw data pointer (mutable)

Returns:

Pointer to tensor data

template<typename T>
inline T *dataPointer() noexcept#

Get typed data pointer (mutable)

Mismatching data type will lead to undefined behavior.

Template Parameters:

T – Data type to cast to

Returns:

Typed pointer to tensor data

template<typename T>
inline T const *dataPointer() const noexcept#

Get typed data pointer (const)

Mismatching data type will lead to undefined behavior.

Template Parameters:

T – Data type to cast to

Returns:

Const typed pointer to tensor data

bool reshape(Coords extent) noexcept#

Reshape the tensor.

Explicitly disallows reshape when the memory is not owned by the tensor object to avoid misuse. Reshape will not happen when memory capacity is insufficient.

Parameters:

extent – New shape

Returns:

True if reshape succeeded, false otherwise

template<typename T>
struct is_arithmetic_ext : public std::is_arithmetic<T>#

Extended arithmetic type trait.

Extends std::is_arithmetic to support fp16 and bfloat16 types for convenience.

Template Parameters:

T – Type to check

template<>
struct is_arithmetic_ext<half> : public std::is_arithmetic<half>, public std::true_type#

Specialization for half precision floating point.

template<>
struct is_arithmetic_ext<__nv_bfloat16> : public std::is_arithmetic<__nv_bfloat16>, public std::true_type#

Specialization for bfloat16 precision floating point.

std::array<int64_t, kMAX_DIMS> trt_edgellm::rt::utils::computeStrides(
Coords const &shape
)#

Compute strides for given shape.

Parameters:

shape – Tensor shape

Returns:

Array of strides

Warning

doxygenfunction: Unable to resolve function “trt_edgellm::rt::utils::getTypeSize” with arguments None in doxygen xml output for project “TensorRT Edge-LLM” from directory: ../cpp_docs/xml. Potential matches: 

- size_t getTypeSize(DataType dataType)
- size_t getTypeSize(nvinfer1::DataType dataType)
std::string trt_edgellm::rt::utils::formatString(
Tensor const &tensor
)#

Format tensor as string for debugging.

Parameters:

tensor – Tensor to format

Returns:

String representation of tensor

double trt_edgellm::rt::utils::toKB(size_t bytes)#

Convert bytes to kilobytes.

Parameters:

bytes – Size in bytes

Returns:

Size in kilobytes

double trt_edgellm::rt::utils::toMB(size_t bytes)#

Convert bytes to megabytes.

Parameters:

bytes – Size in bytes

Returns:

Size in megabytes

double trt_edgellm::rt::utils::toGB(size_t bytes)#

Convert bytes to gigabytes.

Parameters:

bytes – Size in bytes

Returns:

Size in gigabytes