Exchange#

cuda.coop.cutlass.exchange() rearranges register values across a group. It returns a new payload with the requested ownership and preserves the input payload. Exchange itself performs no global-memory load or store.

The API supports blocked/striped, warp-striped, and scatter modes. Use the mode and payload form supported by the selected block or warp group; see cuda.coop.cutlass.exchange().

Striped-to-blocked exchange turns thread-zero values zero and eight into zero and one, with corresponding consecutive pairs for the other threads.

Striped-to-blocked exchange with eight illustrative threads and two items per thread.#

Values retain their logical sequence while their owning threads change.

Thread          0     1     2     3     4     5     6     7
Striped input  0,8   1,9  2,10  3,11  4,12  5,13  6,14  7,15
Blocked result 0,1   2,3   4,5   6,7   8,9  10,11 12,13 14,15

Reading the stages#

See blocked versus striped for a compact ownership table and the index formulas used here.

The first row groups inputs by thread. The middle row shows shared scratch at logical positions; the final row groups the returned values by their new owners. Select a value to inspect its input and result slots.

For K items per thread and T threads, blocked ownership places logical value p in thread p // K, slot p % K. Striped ownership places it in thread p % T, slot p // T. The two blocked/striped modes convert between these arrangements without changing the logical order.

blocked_to_warp_striped and warp_striped_to_blocked apply that conversion separately within each physical warp’s tile. The explorer uses four lanes per teaching warp to fit two warp tiles on screen. Physical CUDA warps have 32 lanes. Warp-striped ownership differs from stripes across a block containing multiple warps.

The scatter modes take a signed-integer ranks payload with the same item count as the input. Each rank selects a logical result position; scatter_to_blocked and scatter_to_striped expose those destinations in their named layouts. The explorer uses rank[p] = (5 * p) % N, a permutation for the illustrated power-of-two tile sizes. Active ranks must be in range and unique.

scatter_to_striped_guarded suppresses inputs with negative ranks; it does not guard against ranks beyond the tile extent. The example assigns rank -1 to every fifth input. scatter_to_striped_flagged instead uses a non-boolean integer valid_flags payload and suppresses every fourth input in the example. Suppressed writes leave holes: a question mark means an unspecified output that must not be consumed, not a zero or a preserved input value.

The illustrations use the default non-timesliced exchange. The block API also provides warp_time_slicing for the layout conversions and unguarded scatters. Guarded and flagged scatter do not support it.

Using Exchange in a kernel#

This CuTe fragment uses the kernel-fragment setup. Launch with 128 threads and provide at least 128 * items_per_thread source and destination elements for each block.

block = coop.this_block()
offset = block_index * 128 * items_per_thread
items = coop.load(
    block, source, items_per_thread=items_per_thread,
    algorithm="striped", offset=offset,
)
blocked = coop.exchange(block, items, mode="striped_to_blocked")
coop.store(block, destination, blocked, algorithm="direct", offset=offset)
# blocked owns consecutive items; items retains striped ownership.
import cutlass
from cutlass import cute

import cuda.coop.cutlass as cutlass_coop

@cute.kernel
def reverse_tile(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = cutlass_coop.this_block()
    thread = block.rank()
    items = cutlass_coop.load(
        block, source, items_per_thread=items_per_thread
    )
    ranks = cutlass_coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        ranks[item] = cutlass.Int32(
            64 * items_per_thread - 1 - (thread * items_per_thread + item)
        )
    reversed_items = cutlass_coop.exchange(
        block, items, mode="scatter_to_blocked", ranks=ranks
    )
    cutlass_coop.store(block, destination, reversed_items)

@cute.jit
def launch(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    reverse_tile(source, destination, items_per_thread).launch(
        grid=1, block=64
    )

See cuda.coop.cutlass.exchange() and the Programming Guide for group participation, scatter controls, and register payload forms.