Logit Bias#
-
struct LogitBias#
Runtime-owned storage for sparse per-request logit biases.
GPU tensors and host staging buffers are allocated once and reused across prefill and vanilla decode steps. An empty full-to-reduced map denotes an identity output vocabulary.
Public Members
-
std::vector<int32_t> hostOffsets#
Reused host staging for CSR offsets.
-
std::vector<int32_t> hostTokenIds#
Reused host staging for token IDs.
-
std::vector<float> hostValues#
Reused host staging for bias values.
-
std::vector<int32_t> fullToReducedVocabMap#
Full token ID to reduced output-vocabulary index.
-
std::vector<int32_t> hostOffsets#
- void trt_edgellm::rt::allocateLogitBias(
- LogitBias &logitBias,
- int32_t maxBatchSize
Allocate reusable logit-bias buffers for a runtime.
- Parameters:
logitBias – Storage to initialize
maxBatchSize – Maximum runtime batch size
- Throws:
std::runtime_error – If maxBatchSize is invalid or tensor allocation fails
- void trt_edgellm::rt::setLogitBiasVocabMap(
- LogitBias &logitBias,
- Tensor const &reducedToFullVocabMap,
- int32_t fullVocabSize,
- int32_t reducedVocabSize,
- cudaStream_t stream
Build the full-to-reduced vocabulary map used when preparing biases.
- Parameters:
logitBias – Storage that owns the inverse map
reducedToFullVocabMap – GPU tensor mapping reduced indices to full token IDs
fullVocabSize – Full tokenizer vocabulary size
reducedVocabSize – Reduced output vocabulary size
stream – CUDA stream used for the device-to-host copy
- Throws:
std::runtime_error – If tensor metadata, vocabulary sizes, token IDs, or CUDA operations are invalid
- bool trt_edgellm::rt::hasLogitBias(
- LLMGenerationRequest const &request
Report whether any slot in a request contains logit-bias entries.
- Parameters:
request – Batched generation request
- Returns:
True when at least one request slot has a non-empty logit-bias map
- std::unordered_map<int32_t, float> trt_edgellm::rt::normalizeLogitBias(
- LogitBias const &logitBias,
- std::unordered_map<int32_t, float> const &fullVocabBias,
- int32_t sequence
Translate one sequence’s sparse full-vocabulary bias into the output vocabulary the sampler indexes.
The identity without a reduced output vocabulary; through the full-to-reduced map otherwise, dropping entries the reduced vocabulary cannot reach (a positive bias dropped this way is logged, naming
sequence). The one translation both the blocking path (prepareLogitBias) and a seated admission apply, so a bias means the same token on both.- Parameters:
logitBias – Runtime-owned vocabulary mapping
fullVocabBias – The request’s bias, keyed by full-vocabulary token id
sequence – Which sequence of the request this is, for diagnostics
- Throws:
std::runtime_error – If a full token ID cannot be mapped safely
- void trt_edgellm::rt::prepareLogitBias(
- LogitBias const &logitBias,
- LLMGenerationRequest const &request,
- DecodingInferenceContext &context
Prepare request-local bias maps in the model output vocabulary.
- Parameters:
logitBias – Runtime-owned vocabulary mapping
request – Validated batched generation request
context – Request-local inference context to populate
- Throws:
std::runtime_error – If a full token ID cannot be mapped safely
- void trt_edgellm::rt::applyLogitBias(
- LogitBias &logitBias,
- Tensor &logits,
- DecodingInferenceContext &context,
- cudaStream_t stream
Apply request logit biases to output logits before sampling.
Dirty request state is flattened and uploaded lazily. Subsequent decode steps reuse the uploaded buffers until batch compaction marks the state dirty.
- Parameters:
logitBias – Runtime-owned GPU buffers and host staging
logits – Base-model output logits to update in place
context – Request-local bias maps and dirty state
stream – CUDA stream used for copies and kernel execution
- Throws:
std::runtime_error – If tensor reshaping, validation, or CUDA operations fail
- void trt_edgellm::rt::applyLogitBiasRepeatedRows(
- LogitBias &logitBias,
- Tensor &logits,
- DecodingInferenceContext &context,
- int32_t rowsPerSlot,
- cudaStream_t stream
Apply request logit biases to repeated per-slot logits rows before sampling or acceptance.
- Parameters:
logitBias – Runtime-owned GPU buffers and host staging
logits – Logits to update in place, shaped [activeBatchSize, rowsPerSlot, vocab] or [activeBatchSize * rowsPerSlot, vocab]
context – Request-local bias maps and dirty state
rowsPerSlot – Number of contiguous logits rows owned by each active request slot
stream – CUDA stream used for copies and kernel execution
- Throws:
std::runtime_error – If tensor reshaping, validation, or CUDA operations fail
- void trt_edgellm::rt::applyLogitBiasForRows(
- LogitBias &logitBias,
- Tensor &logits,
- DecodingInferenceContext &context,
- std::vector<int32_t> const &contextRows,
- cudaStream_t stream
Apply request logit biases to logits rows that sample a selected set of context rows.
Logits row r receives the bias of context row contextRows[r]. A row set equal to the whole active batch in order is applyLogitBias(), including its upload reuse. Any other row set is uploaded on every call and leaves context.logitBiasGpuDirty set, so a later applyLogitBias() re-uploads the all-rows layout. The upload stages through pinned buffers that the next call overwrites, so the caller synchronizes
streambefore calling again.- Parameters:
logitBias – Runtime-owned GPU buffers and host staging
logits – Logits to update in place, shaped [contextRows.size(), vocab]
context – Request-local bias maps and dirty state
contextRows – Distinct rows in [0, context.activeBatchSize), one per logits row
stream – CUDA stream used for copies and kernel execution
- Throws:
std::runtime_error – If the rows or the logits shape are invalid, or CUDA operations fail