Qsa Indexer Jit Compiler

struct QsaIndexerJitKey

Key that identifies one NVRTC-compiled QSA indexer module variant.

One compilation produces a single cubin holding all ten extern “C” entry points (qsa_indexer_q_prep, qsa_indexer_k_compress, qsa_indexer_k_compress_paged, qsa_indexer_scores, qsa_indexer_ids_fill, qsa_indexer_expand, qsa_indexer_raw_k_tail_write_prefill, qsa_indexer_pre_decode, qsa_indexer_scores_decode, qsa_indexer_topk_expand_decode) specialized for (sm, dataType).

Public Functions

inline auto asTuple() const noexcept

inline bool operator==(QsaIndexerJitKey const &other) const noexcept

Public Members

int32_t sm = {}

QsaIndexerJitDataType dataType = {QsaIndexerJitDataType::kHALF}

struct QsaIndexerJitKernel

Public Members

QsaIndexerJitKey key

std::vector<uint8_t> cubin

bool trt_edgellm::canCompileQsaIndexerJitKernel(

QsaIndexerJitKey const &key

) noexcept

QsaIndexerJitKernel trt_edgellm::compileQsaIndexerJitKernel(

QsaIndexerJitKey const &key

)

Compile the QSA indexer kernels for one (sm, dataType) with NVRTC.

Results are shared process-wide through a compile cache keyed by the full key, so repeated calls (including concurrent ones) pay for at most one NVRTC compilation.

Throws:

std::invalid_argument – on an unsupported key, std::runtime_error on NVRTC failure.