Logit Bias#

struct LogitBias#

Runtime-owned storage for sparse per-request logit biases.

GPU tensors and host staging buffers are allocated once and reused across prefill and vanilla decode steps. An empty full-to-reduced map denotes an identity output vocabulary.

Public Members

Tensor tokenIds#

Flattened biased token IDs on GPU.

Tensor values#

Flattened additive bias values on GPU.

Tensor offsets#

Per-slot CSR offsets on GPU.

std::vector<int32_t> hostOffsets#

Reused host staging for CSR offsets.

std::vector<int32_t> hostTokenIds#

Reused host staging for token IDs.

std::vector<float> hostValues#

Reused host staging for bias values.

std::vector<int32_t> fullToReducedVocabMap#

Full token ID to reduced output-vocabulary index.

void trt_edgellm::rt::allocateLogitBias(
LogitBias &logitBias,
int32_t maxBatchSize
)#

Allocate reusable logit-bias buffers for a runtime.

Parameters:

  • logitBias – Storage to initialize

  • maxBatchSize – Maximum runtime batch size

Throws:

std::runtime_error – If maxBatchSize is invalid or tensor allocation fails

void trt_edgellm::rt::setLogitBiasVocabMap(
LogitBias &logitBias,
Tensor const &reducedToFullVocabMap,
int32_t fullVocabSize,
int32_t reducedVocabSize,
cudaStream_t stream
)#

Build the full-to-reduced vocabulary map used when preparing biases.

Parameters:

  • logitBias – Storage that owns the inverse map

  • reducedToFullVocabMap – GPU tensor mapping reduced indices to full token IDs

  • fullVocabSize – Full tokenizer vocabulary size

  • reducedVocabSize – Reduced output vocabulary size

  • stream – CUDA stream used for the device-to-host copy

Throws:

std::runtime_error – If tensor metadata, vocabulary sizes, token IDs, or CUDA operations are invalid

bool trt_edgellm::rt::hasLogitBias(
LLMGenerationRequest const &request
) noexcept#

Report whether any slot in a request contains logit-bias entries.

Parameters:

request – Batched generation request

Returns:

True when at least one request slot has a non-empty logit-bias map

void trt_edgellm::rt::prepareLogitBias(
LogitBias const &logitBias,
LLMGenerationRequest const &request,
DecodingInferenceContext &context
)#

Prepare request-local bias maps in the model output vocabulary.

Parameters:

  • logitBias – Runtime-owned vocabulary mapping

  • request – Validated batched generation request

  • context – Request-local inference context to populate

Throws:

std::runtime_error – If a full token ID cannot be mapped safely

void trt_edgellm::rt::applyLogitBias(
LogitBias &logitBias,
Tensor &logits,
DecodingInferenceContext &context,
cudaStream_t stream
)#

Apply request logit biases to output logits before sampling.

Dirty request state is flattened and uploaded lazily. Subsequent decode steps reuse the uploaded buffers until batch compaction marks the state dirty.

Parameters:

  • logitBias – Runtime-owned GPU buffers and host staging

  • logits – Base-model output logits to update in place

  • context – Request-local bias maps and dirty state

  • stream – CUDA stream used for copies and kernel execution

Throws:

std::runtime_error – If tensor reshaping, validation, or CUDA operations fail

void trt_edgellm::rt::applyLogitBiasRepeatedRows(
LogitBias &logitBias,
Tensor &logits,
DecodingInferenceContext &context,
int32_t rowsPerSlot,
cudaStream_t stream
)#

Apply request logit biases to repeated per-slot logits rows before sampling or acceptance.

Parameters:

  • logitBias – Runtime-owned GPU buffers and host staging

  • logits – Logits to update in place, shaped [activeBatchSize, rowsPerSlot, vocab] or [activeBatchSize * rowsPerSlot, vocab]

  • context – Request-local bias maps and dirty state

  • rowsPerSlot – Number of contiguous logits rows owned by each active request slot

  • stream – CUDA stream used for copies and kernel execution

Throws:

std::runtime_error – If tensor reshaping, validation, or CUDA operations fail