Logit Bias#

struct LogitBias#

Runtime-owned storage for sparse per-request logit biases.

GPU tensors and host staging buffers are allocated once and reused across prefill and vanilla decode steps. An empty full-to-reduced map denotes an identity output vocabulary.

Public Members

Tensor tokenIds#

Flattened biased token IDs on GPU.

Tensor values#

Flattened additive bias values on GPU.

Tensor offsets#

Per-slot CSR offsets on GPU.

std::vector<int32_t> hostOffsets#

Reused host staging for CSR offsets.

std::vector<int32_t> hostTokenIds#

Reused host staging for token IDs.

std::vector<float> hostValues#

Reused host staging for bias values.

Tensor rowSetHostTokenIds#

Pinned staging of applyLogitBiasForRows() token IDs.

Tensor rowSetHostValues#

Pinned staging of applyLogitBiasForRows() bias values.

Tensor rowSetHostOffsets#

Pinned staging of applyLogitBiasForRows() CSR offsets.

std::vector<int32_t> fullToReducedVocabMap#

Full token ID to reduced output-vocabulary index.

void trt_edgellm::rt::allocateLogitBias(
LogitBias &logitBias,
int32_t maxBatchSize
)#

Allocate reusable logit-bias buffers for a runtime.

Parameters:
  • logitBias – Storage to initialize

  • maxBatchSize – Maximum runtime batch size

Throws:

std::runtime_error – If maxBatchSize is invalid or tensor allocation fails

void trt_edgellm::rt::setLogitBiasVocabMap(
LogitBias &logitBias,
Tensor const &reducedToFullVocabMap,
int32_t fullVocabSize,
int32_t reducedVocabSize,
cudaStream_t stream
)#

Build the full-to-reduced vocabulary map used when preparing biases.

Parameters:
  • logitBias – Storage that owns the inverse map

  • reducedToFullVocabMap – GPU tensor mapping reduced indices to full token IDs

  • fullVocabSize – Full tokenizer vocabulary size

  • reducedVocabSize – Reduced output vocabulary size

  • stream – CUDA stream used for the device-to-host copy

Throws:

std::runtime_error – If tensor metadata, vocabulary sizes, token IDs, or CUDA operations are invalid

bool trt_edgellm::rt::hasLogitBias(
LLMGenerationRequest const &request
) noexcept#

Report whether any slot in a request contains logit-bias entries.

Parameters:

request – Batched generation request

Returns:

True when at least one request slot has a non-empty logit-bias map

std::unordered_map<int32_t, float> trt_edgellm::rt::normalizeLogitBias(
LogitBias const &logitBias,
std::unordered_map<int32_t, float> const &fullVocabBias,
int32_t sequence
)#

Translate one sequence’s sparse full-vocabulary bias into the output vocabulary the sampler indexes.

The identity without a reduced output vocabulary; through the full-to-reduced map otherwise, dropping entries the reduced vocabulary cannot reach (a positive bias dropped this way is logged, naming sequence). The one translation both the blocking path (prepareLogitBias) and a seated admission apply, so a bias means the same token on both.

Parameters:
  • logitBias – Runtime-owned vocabulary mapping

  • fullVocabBias – The request’s bias, keyed by full-vocabulary token id

  • sequence – Which sequence of the request this is, for diagnostics

Throws:

std::runtime_error – If a full token ID cannot be mapped safely

void trt_edgellm::rt::prepareLogitBias(
LogitBias const &logitBias,
LLMGenerationRequest const &request,
DecodingInferenceContext &context
)#

Prepare request-local bias maps in the model output vocabulary.

Parameters:
  • logitBias – Runtime-owned vocabulary mapping

  • request – Validated batched generation request

  • context – Request-local inference context to populate

Throws:

std::runtime_error – If a full token ID cannot be mapped safely

void trt_edgellm::rt::applyLogitBias(
LogitBias &logitBias,
Tensor &logits,
DecodingInferenceContext &context,
cudaStream_t stream
)#

Apply request logit biases to output logits before sampling.

Dirty request state is flattened and uploaded lazily. Subsequent decode steps reuse the uploaded buffers until batch compaction marks the state dirty.

Parameters:
  • logitBias – Runtime-owned GPU buffers and host staging

  • logits – Base-model output logits to update in place

  • context – Request-local bias maps and dirty state

  • stream – CUDA stream used for copies and kernel execution

Throws:

std::runtime_error – If tensor reshaping, validation, or CUDA operations fail

void trt_edgellm::rt::applyLogitBiasRepeatedRows(
LogitBias &logitBias,
Tensor &logits,
DecodingInferenceContext &context,
int32_t rowsPerSlot,
cudaStream_t stream
)#

Apply request logit biases to repeated per-slot logits rows before sampling or acceptance.

Parameters:
  • logitBias – Runtime-owned GPU buffers and host staging

  • logits – Logits to update in place, shaped [activeBatchSize, rowsPerSlot, vocab] or [activeBatchSize * rowsPerSlot, vocab]

  • context – Request-local bias maps and dirty state

  • rowsPerSlot – Number of contiguous logits rows owned by each active request slot

  • stream – CUDA stream used for copies and kernel execution

Throws:

std::runtime_error – If tensor reshaping, validation, or CUDA operations fail

void trt_edgellm::rt::applyLogitBiasForRows(
LogitBias &logitBias,
Tensor &logits,
DecodingInferenceContext &context,
std::vector<int32_t> const &contextRows,
cudaStream_t stream
)#

Apply request logit biases to logits rows that sample a selected set of context rows.

Logits row r receives the bias of context row contextRows[r]. A row set equal to the whole active batch in order is applyLogitBias(), including its upload reuse. Any other row set is uploaded on every call and leaves context.logitBiasGpuDirty set, so a later applyLogitBias() re-uploads the all-rows layout. The upload stages through pinned buffers that the next call overwrites, so the caller synchronizes stream before calling again.

Parameters:
  • logitBias – Runtime-owned GPU buffers and host staging

  • logits – Logits to update in place, shaped [contextRows.size(), vocab]

  • context – Request-local bias maps and dirty state

  • contextRows – Distinct rows in [0, context.activeBatchSize), one per logits row

  • stream – CUDA stream used for copies and kernel execution

Throws:

std::runtime_error – If the rows or the logits shape are invalid, or CUDA operations fail