Compact Media Reservation#

class CompactMediaReservation#

Per-slot copies of the encoder outputs that compact multimodal steps embed from.

A prompt’s media rows are produced once at its admission but consumed chunk by chunk over later steps, while later admissions overwrite the encoders’ output buffers. Each resident slot therefore owns a region of E rows in every buffer: the image buffer and one buffer per deepstack layer (E = imageRowsPerSlot) and the audio buffer (E = audioRowsPerSlot). Each buffer is one [slots * E, hiddenSize] FP16 tensor, slot s owning rows [s * E, (s + 1) * E). A region holds, in prompt order, the rows of the media positions at or after the sequence’s prefill start; positions inside a reused prefix are never embedded again.

Public Functions

explicit CompactMediaReservation(Shape const &shape)#

CompactMediaReservation(CompactMediaReservation const&) = delete#: deepstackRows() refers to this object’s own buffers.

CompactMediaReservation &operator=( CompactMediaReservation const& ) = delete#

inline Shape const &shape() const noexcept#

void stage( int32_t slot, std::vector<int32_t> const &promptTokenIds, int32_t prefillStart, int32_t imageTokenId, int32_t audioTokenId, Sources const &sources, cudaStream_t stream )#

Copy into slot's regions the rows of the media positions at or after prefillStart, and record where each prompt position’s row lives. Replaces whatever the slot held.

Throws:: std::runtime_error – when a source does not cover the prompt’s media positions, has the wrong shape or type, or the rows needed exceed the slot’s region.

void release(int32_t slot)#: Forget slot's media; its regions are left as they are and are never read until the next stage().

bool holds(int32_t slot) const#: Whether the sequence on slot was staged with media.

bool mapSlice( int32_t slot, int32_t begin, int32_t length, int32_t *out )#

Write the reservation rows of slot's prompt positions [begin, begin + length) into out, -1 at text positions; the global rows stay below slots * E.

Returns:: Whether any of the positions is a media position.

inline Tensor const &imageRows() const noexcept#: Image rows of every slot, [slots * imageRowsPerSlot, hiddenSize]; empty without a vision encoder.

inline Tensor const &audioRows() const noexcept#: Audio rows of every slot, [slots * audioRowsPerSlot, hiddenSize]; empty without an audio encoder.

inline OptionalInputTensors const &deepstackRows() const noexcept#: Deepstack rows of every slot, one tensor per layer with the image buffer’s shape.

struct Shape#

Public Members

int32_t slots = {0}#

int32_t imageRowsPerSlot = {0}#: 0 without a vision encoder.

int32_t audioRowsPerSlot = {0}#: 0 without an audio encoder.

int32_t hiddenSize = {0}#

int32_t deepstackLayers = {0}#

struct Sources#

One request’s encoder outputs, each [rows, hiddenSize] FP16 with one row per media position of the whole prompt, in prompt order. Only read during stage().

Public Members

Tensor const *image = {nullptr}#

std::vector<std::reference_wrapper<Tensor const>> deepstack#

Tensor const *audio = {nullptr}#

struct Shape

Public Members

int32_t slots = {0}

int32_t imageRowsPerSlot = {0}: 0 without a vision encoder.

int32_t audioRowsPerSlot = {0}: 0 without an audio encoder.

int32_t hiddenSize = {0}

int32_t deepstackLayers = {0}

struct Sources

One request’s encoder outputs, each [rows, hiddenSize] FP16 with one row per media position of the whole prompt, in prompt order. Only read during stage().

Public Members

Tensor const *image = {nullptr}

std::vector<std::reference_wrapper<Tensor const>> deepstack

Tensor const *audio = {nullptr}