Shuffle#
cuda.coop.cutlass.shuffle() shifts values within a complete block and returns a
new per-thread payload. The input remains unchanged. up and down move
one item along the flattened blocked tile, including across thread boundaries.
Scalar offset and rotate modes operate on one value per thread.
See cuda.coop.cutlass.shuffle() for the supported distances.
Up with eight illustrative threads and two items per thread.#
A unit Up shift gives each position its predecessor. ? marks an
undefined output, which must not be read.
Thread 0 1 2 3 4 5 6 7
Input 0,1 2,3 4,5 6,7 8,9 10,11 12,13 14,15
Up ?,0 1,2 3,4 5,6 7,8 9,10 11,12 13,14
Down 1,2 3,4 5,6 7,8 9,10 11,12 13,14 15,?
Reading the stages#
The first row contains working copies of the input. For Up and Down, each thread makes its boundary item available through shared scratch. The other items shift between that thread’s own slots. The last row places the resulting values in blocked ownership.
up reads input[p - 1] for output position p; down reads
input[p + 1]. They support only the compile-time distance 1. The
first Up output and last Down output are undefined. Neither mode wraps or
copies the original boundary value into the result.
The scalar modes use one scalar per thread. offset reads
thread t + distance, with an undefined result when that thread lies
outside the block. rotate reads (t + distance) % block_size and
requires 1 <= distance < block_size. Both accept runtime distances,
which may differ between threads; the explorer uses a uniform distance.
Select a value to follow its source and destination. The shared row shows the values communicated between threads, not a bank layout or a count of hardware instructions.
Using Shuffle in a kernel#
This CuTe fragment uses the kernel-fragment
setup. Launch with 128 threads and provide at
least 128 * items_per_thread source and destination elements per block.
block = coop.this_block()
offset = block_index * 128 * items_per_thread
items = coop.load(
block, source, items_per_thread=items_per_thread, offset=offset
)
shifted = coop.shuffle(block, items, mode="up")
for item in range(items_per_thread):
position = thread_rank * items_per_thread + item
if position > 0:
destination[offset + position] = shifted[item]
Every thread calls Shuffle. The conditional guards only the write afterward, so the undefined first result is never consumed. The first element of each block’s destination tile is left unchanged; initialize those elements separately if they need a value.
from cutlass import cute
import cuda.coop.cutlass as cutlass_coop
@cute.kernel
def rotate_tile(source: cute.Pointer, destination: cute.Pointer):
block = cutlass_coop.this_block()
thread = block.rank()
inputs = cute.make_tensor(source, cute.make_layout(64))
outputs = cute.make_tensor(destination, cute.make_layout(64))
outputs[thread] = cutlass_coop.shuffle(
block, inputs[thread], mode="rotate", distance=7
)
@cute.jit
def launch(source: cute.Pointer, destination: cute.Pointer):
rotate_tile(source, destination).launch(grid=1, block=64)
See cuda.coop.cutlass.shuffle() and
the Programming Guide for scalar and array modes.