Decoding Inference Context#

struct LogprobsSlot#

Pre-allocated flat accumulator for per-step log-probabilities of one batch slot.

Stores up to maxGenerateLength * topK pairs in a single allocation made at request start, eliminating per-step heap allocations inside the decode loop. data[step * topK .. (step+1) * topK - 1] holds the top-K (token_id, log_prob) pairs for step step.

Public Members

std::vector<std::pair<int32_t, float>> data#: Flat [maxGenerateLength * topK] storage, pre-allocated once.

int32_t numSteps = {0}#: Number of steps written so far.

struct BatchResult#

Batch result data for a single sequence.

Encapsulates all data needed to track a batch’s execution results, whether the sequence is still active or has already been evicted.

Public Members

std::vector<int32_t> tokenIds#: Generated token IDs.

std::vector<int32_t> rawBatchedInputIds#: Original input token IDs.

int32_t generateLength = {0}#: Number of tokens generated.

int32_t actualIterations = {0}#: Number of iterations executed.

int32_t effectivePrefillLength = {0}#: Effective prefill length after system prompt cache reuse

std::vector<std::vector<LogprobEntry>> logprobs#: Per-step top log-probabilities: logprobs[step] = [LogprobEntry, …], sorted descending. Populated only when numLogprobs > 0 in the original request.

FinishReason terminalReason{FinishReason::kNotFinished}#: Why this batch terminated (EOS, length, stop string, cancel, error)

struct DecodingInferenceContext#

Per-request execution context shared by runtime and decoding strategies.

Holds request-local sequence metadata, sampling parameters, multimodal embedding references, streaming state, and batch-eviction bookkeeping.

Public Functions

void initialize( int32_t batchSize, int32_t maxGenLength, rt::OptionalInputTensor const &visual, rt::OptionalInputTensors const &deepstackFeatures, std::string const &loraName, cudaStream_t cudaStream, )#

Initialize request-local vectors and scalar fields.

Parameters:

batchSize – Active batch size
maxGenLength – Maximum generation length
visual – Optional visual embeddings
deepstackFeatures – Deepstack features for Qwen3-VL
loraName – LoRA weights name used by this request
cudaStream – CUDA stream for operations

DecodingInferenceContext()#: ctor / move ops / dtor are out-of-line (defined in the .cpp) because layerDebugger is a unique_ptr to the incomplete type LayerDebugger: a defaulted special member in the header would instantiate the member’s destructor against the incomplete type in every translation unit (e.g. unit tests that only forward-declare LayerDebugger). Move ops are declared because the user-declared destructor otherwise suppresses the implicit move, and the context is returned by value in places. The struct stays non-copyable via the unique_ptr member.

DecodingInferenceContext(DecodingInferenceContext&&) noexcept#

DecodingInferenceContext &operator=( DecodingInferenceContext&&, ) noexcept#

~DecodingInferenceContext()#

Public Members

std::vector<std::string> systemPrompts#: System prompts for each sequence in batch.

std::vector<std::vector<int32_t>> rawBatchedInputIds#: Original token IDs before preprocessing.

std::vector<std::vector<int32_t>> tokenIds#: Token IDs for each sequence: [batch_size][seq_length].

std::vector<int32_t> currentGenerateLengths#: Current generation length for each sequence.

std::vector<int32_t> effectivePrefillLengths#: Prefill length after system prompt cache reuse.

std::vector<int8_t> finishedStates#: Finished state for each sequence.

std::unordered_map<int32_t, BatchResult> completedBatches#: Results of completed batches.

std::vector<int32_t> batchIndexMapping#: Maps current batch index to original index.

std::vector<SlotStreamState> slotStreams#: Per-slot streaming state.

rt::OptionalInputTensor visualEmbeddings#: Optional visual embeddings.

rt::OptionalInputTensor audioEmbeddings#: Optional audio embeddings.

rt::OptionalInputTensors deepstackFeatures#: Optional Deepstack features.

int32_t generationRound = {}#: Current generation round.

int32_t maxGenerateLength = {}#: Maximum generation length.

int32_t activeBatchSize = {}#: Current active batch size.

std::string loraWeightsName = {""}#: LoRA adapter name used by this request.

cudaStream_t stream = {}#: CUDA stream.

float temperature = {1.0f}#: Temperature for sampling.

float topP = {1.0f}#: Top-P sampling parameter.

int64_t topK = {0}#: Top-K sampling parameter.

int32_t numLogprobs = {0}#: Number of top log-probs to collect per generated token

std::vector<rt::LogprobsSlot> stepLogprobs#: Per-batch flat logprobs accumulator. slot.data is pre-allocated [(maxGenerateLength + draftingStep) * numLogprobs] in spec-decode mode (vanilla: maxGenerateLength) to accommodate the up-to-(draftingStep+1) tokens accepted per verify step. slot.data[step*numLogprobs .. (step+1)*numLogprobs-1] holds step’s top-K (token_id, log_prob) pairs.

std::vector<std::vector<std::string>> stopStringsPerSlot#

std::vector<std::unordered_map<int32_t, float>> logitBiasPerSlot#: Per-active-slot sparse logit bias maps in output-vocab space.

bool hasLogitBias = {false}#: True when any active slot has logit bias entries.

bool logitBiasGpuDirty = {false}#: True when CPU-side bias state must be uploaded to GPU.

bool outputThinkerEmbeddings = {false}#: Whether to capture hidden states for the Talker pipeline.

std::optional<TokenCallback> onTokenGenerated#: Optional per-token callback invoked after each accepted token update.

std::function<bool(int32_t, int32_t)> shouldStopAfterAcceptedToken#: Optional callback used by speculative decoders to stop appending accepted tokens.

std::unique_ptr<LayerDebugger> layerDebugger#: Few-layer-validation debug: per-request layer dumper (null unless the EDGELLM_DUMP_LOGITS_KVCACHE_* env vars are set). Owned here via RAII so it shares the context’s lifetime exactly; see the out-of-line destructor. Also carries optional teacher-forcing tokens (EDGELLM_FORCE_TOKENS_FILE) applied via LayerDebugger::applyForcedTokens.