Programming concepts#

Cooperative operations let a group of threads work on data together. This page explains how threads divide a tile, which threads must participate, and how to use results and temporary storage. The Programming Guide provides complete CuTe kernels and launch examples.

Kernel API#

Use import cuda.coop.cutlass as coop. Calls inside a CuTe kernel are compiler markers; compilation turns them into device operations. The cuda.coop parent is a namespace package. The standalone distribution installs its public operations only under cuda.coop.cutlass.

Built-in operators and their documented aliases are supported. Reduce and Scan also accept compiled scalar device operators. Prepare these operators on the host before kernel tracing. Stateful Scan prefix callbacks are unsupported.

Calling conventions#

A primitive’s group and input/output operands precede / in its signature and are passed positionally. Controls after * are keyword-only:

items = coop.load(
    group, source, items_per_thread=items_per_thread,
    valid_items=count, offset=offset,
)
result = coop.reduce(group, items, binary_op="max")

Load always returns a fresh ThreadData payload. An operation that returns a new value leaves its input payload unchanged unless its contract says otherwise. Check cuda.coop.cutlass API Reference for each operation’s accepted operands, controls, and ownership rules.

Shared execution model#

Groups and tiles#

A group identifies the threads working together. this_block() selects the current block; this_warp() selects a physical 32-thread warp. this_warp().group_by(width) partitions it into consecutive logical warps of 1, 2, 4, 8, 16, or 32 threads. A group processing K items per thread has a tile of group_size * K items. Each block or warp group processes its own tile.

The hierarchy also describes individual threads, clusters, the grid, and mapped groups of physical warps. These descriptors support queries such as rank, count, and membership. Their availability does not make every primitive valid for that group: Reduce, Load/Store, and Scan use block or warp groups, and grid primitives are unavailable. Consult the backend guide for the supported query levels and synchronization operations.

A multidimensional block uses x-major linear thread rank: x + block_x * (y + block_y * z). Warp primitives require an enclosing block made of complete physical warps. For warp Load/Store, the compiler adds the group’s within-block tile origin to the memory address. A caller’s offset supplies the remaining element offset, such as the block’s tile origin in a larger array; do not add the within-block group origin again.

Participation and valid prefixes#

Every required participant must reach the same primitive invocation. A short tile does not reduce the number of threads that must participate. For example, valid_items=45 limits a 64-item Load to its first 45 tile positions; all threads in the group still call Load. Complete sibling logical warps may take different paths where the operation’s contract permits it.

For Load and Store, valid_items counts tile elements, not items per thread. It must be uniform within the group and lie between zero and the tile capacity. Clamp a tail count before passing it. Invalid static counts are rejected during compilation; invalid runtime counts trap on the device. offset is a nonnegative element offset, also uniform within the group. It is independent of the valid count. Other families define their own valid-prefix controls; for example, reduction counts contributing threads.

Load leaves invalid payload slots unspecified unless oob_default is provided, even if those slots were initialized before Load. Store leaves destination elements outside the valid prefix untouched. A valid prefix controls data access; it does not make it safe to skip a required participant or a reuse barrier.

Per-thread payloads#

Prefer coop.load(group, source, items_per_thread=items_per_thread) for each participating thread’s fixed-size ThreadData payload. Its element type comes from source. Pass the positive compile-time count as a kernel argument declared items_per_thread: cutlass.Constexpr in both the CuTe kernel and its launcher. A count of one still returns ThreadData.

Use coop.ThreadData(items_per_thread) to construct a payload for computed values or writable auxiliary outputs. Dtype-free construction remains supported: initialize every consumed item with a compatible typed value. Load’s items_per_thread and alignment are keyword-only controls.

Global and shared arrays can supply Load’s source and receive Store’s output. Those arrays hold a group tile; a ThreadData holds one thread’s values. Use native CuTe copies to stage a tile directly in shared memory, then synchronize before another thread consumes it. See shared-memory staging.

Supported payload dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers and 32- and 64-bit floating-point values. An explicit alignment is a positive power of two in bytes and sets a minimum when payload storage is materialized. It does not assert alignment of an input or output array. For example, coop.load(group, source, items_per_thread=items_per_thread, alignment=16) requests at least 16-byte alignment for the returned payload, as does coop.ThreadData(items_per_thread, alignment=16) for a directly constructed payload. None lets the compiler choose; it may use stronger alignment.

The payload extent is a compile-time value available as items.items_per_thread. Directly constructed slots start uninitialized. Initialize every item an operation will read; Load without oob_default leaves invalid slots unspecified. Assign those slots after Load before reading them, or supply a default. As in CUB, transpose Store algorithms may rearrange the input payload in place. Copy values before Store if they are needed later.

Typing protocols describe the supported payload interfaces; protocol compatibility alone does not make an arbitrary Python object a kernel value. CuTe register tensors and immutable TensorSSA values have explicit adapters. Typed assignments and supported producers can establish the element type. See element-type inference and register and producer conversions for ownership, lifetime, and runtime-control-flow rules.

Layouts and operation order#

In a blocked layout, each thread owns consecutive tile positions. In a striped layout, consecutive threads own consecutive positions at each item slot. The layout glossary gives the formulas and an ownership table.

Load/Store’s direct and vectorize algorithms expose blocked payloads. striped exposes striped payloads. Transpose algorithms use striped memory accesses and present blocked payloads to the caller. Exchange converts between supported layouts. ThreadData does not carry a layout tag that automatically fixes a mismatched Load/Store pair. Shuffle shifts values within a block’s tile. The Exchange and Shuffle visualizations show the supported rearrangements.

Scan traverses the group’s values in blocked tile order. A striped Load must therefore be converted before a Scan intended to follow the array’s original order. Choosing a memory-access algorithm and choosing the order of values seen by an operation are related but distinct decisions.

Result ownership#

A primitive’s return type does not say which threads may use its result. Reduce/Sum define their result only at group rank zero. Scan returns each thread’s part of the prefix sequence. Check the operation’s group and algorithm contract before consuming a result.

Sorting and selection operate on one group’s tile. Sorting each block does not sort a whole array. TopK defines an unordered selected prefix; the remaining payload positions are not output. See the API reference for sorting and selection operations.

Scratch allocation and reuse#

The compiler allocates temporary shared storage for operations that need it. Direct, striped, and vectorized Load/Store are storage-free: they add no scratch pointer, shared allocation, or reuse barrier. Other algorithms use the concrete CUB specialization’s required size and alignment.

For operations that accept an explicit descriptor, construct TempStorage inside the kernel. A descriptor lets several calls reuse one region or request capacity and minimum alignment. An undersized request is an error. sharing="shared" overlaps the uses of one descriptor; sharing="exclusive" assigns distinct call sites separate slices. With automatic synchronization disabled, separate slices avoid barriers needed solely for cross-call scratch reuse, at the cost of more shared memory. Independent descriptors do not alias. See the exclusive-storage FAQ.

Explicit descriptors default to auto_sync=False for both sharing modes. The kernel must provide the required barrier before reuse, including across iterations: a call site inside a loop reuses its slice even with exclusive storage. Set auto_sync=True to insert a trailing reuse barrier after each storage-using call. Compiler-managed scratch, used when no descriptor is supplied, synchronizes automatically. A scratch reuse barrier does not replace synchronization for the kernel’s own shared data.

Warp operations that need scratch keep independent storage per physical or logical group and use the appropriate warp mask. Each primitive documents whether it accepts explicit storage. See CUTLASS storage and the storage FAQ for examples and limits.

reserve(num_elems, dtype, alignment=...) returns a typed shared view from a descriptor. Shared policy may alias primitive scratch; exclusive policy assigns slices. A reservation is application data, so a scratch-reuse barrier alone does not establish that another thread can consume its contents.

release() consumes the descriptor and its aliases on the executing path and exposes the entire existing region, including padding and exclusive slices. It inserts no barrier or wait. Finish outstanding uses and synchronize as required before release; subsequent descriptor or stale-alias use is an error. Conditional and repeated execution must respect the same lifetime.