Ragged Batch Builder#
-
class IdentitySetScratch#
-
class RaggedBatchBuilder#
Lowers a ScheduledStep according to the contract backend.
Entry-padded: every sequence occupies queryWidth physical rows; context and decode work cannot mix. Compact (phase 8): the step must list every context sequence before every decode sequence. Execution row i is step.sequences[i]; context rows [0, Np) own T_p tokens, decode rows [Np, N) own one token each, and tokens are packed without padding (T = T_p + N - Np). logitsToSequence maps each logits row to its execution row.
Public Functions
-
explicit RaggedBatchBuilder(RaggedEngineContract contract)#
-
void reserve(RaggedExecutionBatch &batch)#
- void buildInto(
- ScheduledStep const &step,
- RaggedExecutionBatch &batch
- void finalizeSubsetSelection(
- ScheduledStep const &sourceStep,
- std::vector<int32_t> const &keepStartOffsets,
- std::vector<int32_t> const &concatenatedKeepIndices,
- RaggedExecutionBatch &batch
Public Static Functions
- static void validateRuntimeAdapterStep(
- ScheduledStep const &step,
- int32_t activeBatchSize,
- std::vector<RequestId> const &requestIds,
- std::vector<ResidentRef> const &residentRefs
- static void validateExecutionBatch(
- RaggedExecutionBatch const &batch,
- RaggedEngineContract const &contract
-
explicit RaggedBatchBuilder(RaggedEngineContract contract)#
-
struct RaggedEngineContract#
The entry-padded backend requires mixedStepSupported == false; the compact backend requires it to be true.
-
struct RaggedStepShape#
-
struct CompletionSequence#
-
struct RaggedExecutionBatch#
Public Functions
- void validateCommitSnapshot(
- StepId completedStepId,
- std::vector<CompletionSequence> const ¤t
Public Members
-
StepId stepId = {0}#
-
TokenLayoutBackend layout = {TokenLayoutBackend::kEntryPaddedCompatibility}#
-
RaggedStepShape shape#
-
std::vector<SequenceIdentity> sequenceOrder#
-
std::vector<int32_t> positions#
-
std::vector<int32_t> queryStartOffsets#
-
std::vector<int32_t> queryLengths#
-
std::vector<int32_t> pastLengths#
-
std::vector<int32_t> attentionSequenceLengths#
-
std::vector<int32_t> stateIndices#
-
std::vector<int64_t> logitsIndices#
-
std::vector<int32_t> logitsToSequence#
-
std::vector<SequenceWork> sequenceWorks#
-
int32_t placeholderLogitsRows = {0}#
Compact layout only: logits rows after the logitsToSequence.size() routed ones, one per sequence that selects no logits (its last token row), in execution order. The exported graph sizes logits_indices by num_sequences, so a compact step always has L = N; the engine computes these rows and they are discarded.