Decoder XQA Jit Compiler#

struct XQAJitKey#

Key that identifies one NVRTC-compiled XQA decode kernel variant. Follows the plugin JIT key contract in pluginJitKey.h.

Public Functions

inline std::array<uint32_t, kNumWords> keyWords() const noexcept#

The fields in blob, digest and hash order.

inline bool operator==(XQAJitKey const &other) const noexcept#

Public Members

int32_t sm = {}#
trt_edgellm::DataType dataType = {trt_edgellm::DataType::kHALF}#
trt_edgellm::DataType kvDataType = {trt_edgellm::DataType::kHALF}#
int32_t headSize = {}#
int32_t qHeadsPerKv = {}#
int32_t tokensPerPage = {}#
bool slidingWindow = {}#
bool specDecode = {}#
bool contiguousQuerySwa = {}#

Public Static Functions

static inline XQAJitKey fromKeyWords(
std::array<uint32_t, kNumWords> const &words
) noexcept#

Public Static Attributes

static constexpr size_t kNumWords = {9}#
std::vector<uint8_t> trt_edgellm::serializeXQAJitKernels(
std::vector<XQAJitKernel> const &kernels
)#

Serialize compiled XQA kernels into a self-describing byte blob.

The key is serialized field by field alongside its cubin so that a deserialized kernel is registered under the key it was actually compiled for, rather than one recomputed from whatever plugin fields happen to be present at deserialization time.

std::vector<XQAJitKernel> trt_edgellm::deserializeXQAJitKernels(
void const *data,
size_t size
)#

Inverse of serializeXQAJitKernels.

Throws:

std::runtime_error – if the blob is truncated, tampered with, or does not open with the XQA magic word.

XQAJitKernel trt_edgellm::compileXQAKernel(XQAJitKey const &key)#

Compile one XQA kernel variant with NVRTC.

Throws:

std::runtime_error – if NVRTC support is disabled or compilation fails.