Batched Warp Reduction#
cuda.coop.cutlass.reduce_batched() reduces several independent batches across
a warp. Each thread contributes one item to each batch: local slot zero
belongs to batch zero, slot one to batch one, and so on. The result has
one aggregate per batch, distributed among the threads.
This is useful when every lane holds several features for one sample and
the kernel needs a sum, minimum, or maximum for each feature. Ordinary
reduce() combines a group’s payload items into one
aggregate; batched reduction retains the separate batch axis.
Three batches across a logical warp of four lanes.#
Each column below is one independent reduction. ? is an unspecified
output slot, which the kernel must not read or store.
Lane Input slots (batches 0, 1, 2) Returned slot
L0 1 2 3 16
L1 3 4 5 20
L2 5 6 7 24
L3 7 8 9 ?
Batch sums 16 20 24
Input slots and output ownership#
For B batches and a warp of W threads, each input payload contains
B items. Each returned payload contains ceil(B / W) slots.
output_layout="striped" assigns batch lane + slot * W to each
result slot. output_layout="blocked" assigns batch
lane * ceil(B / W) + slot instead. The input interpretation is the
same for both layouts.
Try ten batches with four lanes to see the distinction: striped lane zero receives batches 0, 4, and 8; blocked lane zero receives batches 0, 1, and 2. The remaining two allocated output slots do not correspond to any batch. Guard stores by batch index, rather than assuming every slot in the returned payload is valid.
The explorer groups inputs by batch to explain the reduction axis. It
does not imply that CUB performs a shared-memory transpose. CUB provides
a native WarpReduceBatched collective; the diagrams describe its
result contract, not its compiled instruction order.
Using batched reduction in a kernel#
This tested example sums items_per_thread features across each physical
warp. The input is laid out as 32 consecutive samples per warp, with
items_per_thread consecutive features per sample. Each warp writes one
result per feature.
@cute.kernel
def feature_sums(
samples: cute.Pointer,
totals: cute.Pointer,
items_per_thread: cutlass.Constexpr,
):
warp = coop.this_warp()
features = coop.load(warp, samples, items_per_thread=items_per_thread)
sums = coop.reduce_batched(warp, features)
outputs = cute.make_tensor(
totals, cute.make_layout(2 * items_per_thread)
)
# Output slots are striped across the warp lanes.
warp_index = coop.this_block().rank() // 32
for item in cutlass.range_constexpr((items_per_thread + 31) // 32):
feature = warp.rank() + item * 32
if feature < items_per_thread:
outputs[warp_index * items_per_thread + feature] = sums[item]
@cute.jit
def launch(
samples: cute.Pointer,
totals: cute.Pointer,
items_per_thread: cutlass.Constexpr,
):
feature_sums(samples, totals, items_per_thread).launch(
grid=1, block=(8, 4, 2)
)
The API supports complete physical warps and logical warps of 1, 2, 4, 8,
16, or 32 threads. Every member of the selected warp must participate, even if
that member owns no result. Other logical warps may take another branch. The
compiler manages provider storage; this operation does not accept a caller
TempStorage. The CUTLASS provider needs no shared scratch allocation or
trailing storage-reuse barrier.
The batch count is a positive compile-time payload extent. The result
uses the input dtype without accumulator promotion. Built-in operator
strings match reduce(). CuTe register tensors and
immutable vectors are accepted along with ThreadData. The operation
preserves its input and returns a fresh payload. Use only the result slots
whose batch index is in range; no temporary-storage descriptor is needed.