Generation Boundary#

class GenerationBoundary#

Subclassed by trt_edgellm::rt::LLMRankRuntime::GenerationSession

Public Functions

virtual AdmitDecision admitSequence(SlotSeed seed) = 0#

Join one new sequence to the running batch and prefill it.

Throws:

std::runtime_error – for seeds the batch rejects outright; nothing is modified then.

virtual AdmitDecision admitRequest(
LLMGenerationRequest const &request,
int32_t originalIndex,
RequestId requestId
) = 0#

Admit a caller-level request: tokenization included, so a scheduler never builds a SlotSeed by hand or holds a tokenizer.

Whether the request may share this batch at all — sampling parameters, adapter, budgets — is the scheduler’s question (BatchCompatibility), answered before calling this.

Parameters:

request – A single-sequence request. Its stream channel, stop strings and logit bias ride along; its sampling parameters are ignored in favour of the batch’s. Scheduler policy keeps media requests founder-only until live admission has dedicated coverage.

Throws:

std::runtime_error – for an invalid request or an unsupported media deployment; nothing is modified.

virtual LLMGenerationResponse materializeResult(
BatchResult const &result,
std::vector<std::string> const &stopStrings
) const = 0#

Turn one finished sequence’s result into the caller-facing response, decoding included, exactly as the founding request’s own assembly would have.

Lives here because the conversion needs the tokenizer and the stop-string trimming that the runtime owns; a scheduler holding raw BatchResults would either skip the text or grow its own copy of both.

virtual std::unordered_map<int32_t, BatchResult> takeCompletedAtOrAbove(
int32_t firstIndex
) = 0#

Move out results of finished sequences whose original index is >= firstIndex.

Sequences admitted mid-flight carry indices above the founding request’s range; harvesting them at each boundary is what keeps them out of the founding caller’s response.

virtual int32_t residentCount() const = 0#

Sequences currently resident, finished or not. Admission capacity is judged on this.