Decoder XQA Jit Compiler#
-
struct XQAJitKey#
Key that identifies one NVRTC-compiled XQA decode kernel variant. Follows the plugin JIT key contract in pluginJitKey.h.
Public Functions
Public Members
-
int32_t sm = {}#
-
trt_edgellm::DataType dataType = {trt_edgellm::DataType::kHALF}#
-
trt_edgellm::DataType kvDataType = {trt_edgellm::DataType::kHALF}#
-
int32_t headSize = {}#
-
int32_t qHeadsPerKv = {}#
-
int32_t tokensPerPage = {}#
-
bool slidingWindow = {}#
-
bool specDecode = {}#
-
bool contiguousQuerySwa = {}#
Public Static Functions
Public Static Attributes
-
static constexpr size_t kNumWords = {9}#
-
int32_t sm = {}#
- std::vector<uint8_t> trt_edgellm::serializeXQAJitKernels(
- std::vector<XQAJitKernel> const &kernels
Serialize compiled XQA kernels into a self-describing byte blob.
The key is serialized field by field alongside its cubin so that a deserialized kernel is registered under the key it was actually compiled for, rather than one recomputed from whatever plugin fields happen to be present at deserialization time.
- std::vector<XQAJitKernel> trt_edgellm::deserializeXQAJitKernels(
- void const *data,
- size_t size
Inverse of serializeXQAJitKernels.
- Throws:
std::runtime_error – if the blob is truncated, tampered with, or does not open with the XQA magic word.