Logit Bias#

struct LogitBias#

Runtime-owned storage for sparse per-request logit biases.

GPU tensors and host staging buffers are allocated once and reused across prefill and vanilla decode steps. An empty full-to-reduced map denotes an identity output vocabulary.

Public Members

Tensor tokenIds#

Flattened biased token IDs on GPU.

Tensor values#

Flattened additive bias values on GPU.

Tensor offsets#

Per-slot CSR offsets on GPU.

std::vector<int32_t> hostOffsets#

Reused host staging for CSR offsets.

std::vector<int32_t> hostTokenIds#

Reused host staging for token IDs.

std::vector<float> hostValues#

Reused host staging for bias values.

std::vector<int32_t> fullToReducedVocabMap#

Full token ID to reduced output-vocabulary index.

void trt_edgellm::rt::allocateLogitBias(
LogitBias &logitBias,
int32_t maxBatchSize,
)#

Allocate reusable logit-bias buffers for a runtime.

Parameters:

  • logitBias – Storage to initialize

  • maxBatchSize – Maximum runtime batch size

Throws:

std::runtime_error – If maxBatchSize is invalid or tensor allocation fails

void trt_edgellm::rt::setLogitBiasVocabMap(
LogitBias &logitBias,
Tensor const &reducedToFullVocabMap,
int32_t fullVocabSize,
int32_t reducedVocabSize,
cudaStream_t stream,
)#

Build the full-to-reduced vocabulary map used when preparing biases.

Parameters:

  • logitBias – Storage that owns the inverse map

  • reducedToFullVocabMap – GPU tensor mapping reduced indices to full token IDs

  • fullVocabSize – Full tokenizer vocabulary size

  • reducedVocabSize – Reduced output vocabulary size

  • stream – CUDA stream used for the device-to-host copy

Throws:

std::runtime_error – If tensor metadata, vocabulary sizes, token IDs, or CUDA operations are invalid

bool trt_edgellm::rt::hasLogitBias(
LLMGenerationRequest const &request,
) noexcept#

Report whether any slot in a request contains logit-bias entries.

Parameters:

request – Batched generation request

Returns:

True when at least one request slot has a non-empty logit-bias map

bool trt_edgellm::rt::shouldRejectLogitBiasWithSpecDecode(
LLMGenerationRequest const &request,
bool speculativeDecoderAvailable,
) noexcept#

Return true when logit bias must be rejected for an active speculative decoder.

Logit bias remains valid when speculative decoding is unavailable or the caller explicitly disables it.

Parameters:

  • request – Batched generation request

  • speculativeDecoderAvailable – Whether the runtime has a speculative decoder

void trt_edgellm::rt::prepareLogitBias(
LogitBias const &logitBias,
LLMGenerationRequest const &request,
DecodingInferenceContext &context,
)#

Prepare request-local bias maps in the model output vocabulary.

Parameters:

  • logitBias – Runtime-owned vocabulary mapping

  • request – Validated batched generation request

  • context – Request-local inference context to populate

Throws:

std::runtime_error – If a full token ID cannot be mapped safely

void trt_edgellm::rt::applyLogitBias(
LogitBias &logitBias,
Tensor &logits,
DecodingInferenceContext &context,
cudaStream_t stream,
)#

Apply request logit biases to output logits before sampling.

Dirty request state is flattened and uploaded lazily. Subsequent decode steps reuse the uploaded buffers until batch compaction marks the state dirty.

Parameters:

  • logitBias – Runtime-owned GPU buffers and host staging

  • logits – Base-model output logits to update in place

  • context – Request-local bias maps and dirty state

  • stream – CUDA stream used for copies and kernel execution

Throws:

std::runtime_error – If tensor reshaping, validation, or CUDA operations fail