Shuffle#

cuda.coop.cutlass.shuffle() shifts values within a complete block and returns a new per-thread payload. The input remains unchanged. up and down move one item along the flattened blocked tile, including across thread boundaries. Scalar offset and rotate modes operate on one value per thread. See cuda.coop.cutlass.shuffle() for the supported distances.

Up shuffle moves each input one position to the right in blocked order; the first output has no source and is undefined.

Up with eight illustrative threads and two items per thread.#

A unit Up shift gives each position its predecessor. ? marks an undefined output, which must not be read.

Thread     0     1     2     3     4     5      6      7
Input     0,1   2,3   4,5   6,7   8,9  10,11  12,13  14,15
Up        ?,0   1,2   3,4   5,6   7,8   9,10  11,12  13,14
Down      1,2   3,4   5,6   7,8  9,10  11,12  13,14   15,?

Reading the stages#

The first row contains working copies of the input. For Up and Down, each thread makes its boundary item available through shared scratch. The other items shift between that thread’s own slots. The last row places the resulting values in blocked ownership.

up reads input[p - 1] for output position p; down reads input[p + 1]. They support only the compile-time distance 1. The first Up output and last Down output are undefined. Neither mode wraps or copies the original boundary value into the result.

The scalar modes use one scalar per thread. offset reads thread t + distance, with an undefined result when that thread lies outside the block. rotate reads (t + distance) % block_size and requires 1 <= distance < block_size. Both accept runtime distances, which may differ between threads; the explorer uses a uniform distance.

Select a value to follow its source and destination. The shared row shows the values communicated between threads, not a bank layout or a count of hardware instructions.

Using Shuffle in a kernel#

This CuTe fragment uses the kernel-fragment setup. Launch with 128 threads and provide at least 128 * items_per_thread source and destination elements per block.

block = coop.this_block()
offset = block_index * 128 * items_per_thread
items = coop.load(
    block, source, items_per_thread=items_per_thread, offset=offset
)
shifted = coop.shuffle(block, items, mode="up")
for item in range(items_per_thread):
    position = thread_rank * items_per_thread + item
    if position > 0:
        destination[offset + position] = shifted[item]

Every thread calls Shuffle. The conditional guards only the write afterward, so the undefined first result is never consumed. The first element of each block’s destination tile is left unchanged; initialize those elements separately if they need a value.

from cutlass import cute

import cuda.coop.cutlass as cutlass_coop

@cute.kernel
def rotate_tile(source: cute.Pointer, destination: cute.Pointer):
    block = cutlass_coop.this_block()
    thread = block.rank()
    inputs = cute.make_tensor(source, cute.make_layout(64))
    outputs = cute.make_tensor(destination, cute.make_layout(64))
    outputs[thread] = cutlass_coop.shuffle(
        block, inputs[thread], mode="rotate", distance=7
    )

@cute.jit
def launch(source: cute.Pointer, destination: cute.Pointer):
    rotate_tile(source, destination).launch(grid=1, block=64)

See cuda.coop.cutlass.shuffle() and the Programming Guide for scalar and array modes.