Tensor Parallel Plugin Resources#

class TensorParallelPluginResources : public trt_edgellm::rt::MultiDevicePluginResources #

Own and register communication resources for single-process TP launch.

Threaded TP launch initializes all NCCL communicators in one process through ncclCommInitAll(), then registers the per-device communicator handles with the TensorRT plugin registry before rank-local engines are deserialized.

Public Functions

explicit TensorParallelPluginResources( TensorParallelPluginResourcesConfig const &config )#

~TensorParallelPluginResources() noexcept override#

TensorParallelPluginResources( TensorParallelPluginResources const& ) = delete#

TensorParallelPluginResources &operator=( TensorParallelPluginResources const& ) = delete#

TensorParallelPluginResources( TensorParallelPluginResources&& ) = delete#

TensorParallelPluginResources &operator=( TensorParallelPluginResources&& ) = delete#

inline virtual ParallelType type() const noexcept override#

inline virtual int32_t size() const noexcept override#

virtual RuntimeCollectiveResources const *runtimeCollectives( ) const noexcept override#

virtual bool hasAllReducePath( AllReducePathType type ) const noexcept override#

virtual bool abortOwnedRuntimeCollectives() noexcept override#

struct TensorParallelPluginResourcesConfig#

Configuration for single-process TP plugin communication resources.

Public Members

int32_t tpSize = {1}#: Tensor parallel world size.

std::vector<int32_t> localRanks#: TP ranks owned by this process.

std::vector<int32_t> localDevices#

std::vector<void*> ncclComms#

bool ownsNcclComms = {true}#: Transfer ownership of non-empty externally supplied NCCL communicators.