cuda.coop.cutlass API Reference#

Warning

This is an experimental API and is subject to change.

Use import cuda.coop.cutlass as coop inside programs compiled with CuTe. The installed .pyi files describe overload and result typing. The Programming Guide explains payload ownership, participation, layouts, and operation-specific controls.

Load always returns a fresh ThreadData, including when items_per_thread=1. Global and shared tensors can be Load sources or Store destinations. CuTe register and immutable-vector adapters provide the conversion rules for operations accepting those forms.

Reduce and Sum return CuTe scalars, even when a NumPy dtype selects the payload type; stubs report that NumPy type for static checks. Array operations return ThreadData unless their documented contract says otherwise. Warp Scan accepts one scalar per lane. Rank results and Discontinuity flags have signed Int32 dtype. Bulk Run Length Decode returns a Uint32 total to every block member.

Recognized operator functions and NumPy ufuncs select built-in operators. For custom Reduce and Scan binary operators, prepare a DeviceOperator with compile_operator() before tracing. See Compiled scalar device operators for the scalar signature and compiler requirements. Scan prefix callbacks and runtime callback state are unsupported.

Compiled device operators#

cuda.coop.cutlass.compile_operator(
func: Any,
dtype: Any,
*,
arch: str | None = None,
) → DeviceOperator#

Compile a stateless @cute.jit binary operator before kernel tracing.

Parameters:
  • func (callable) – A @cute.jit function taking two scalar operands and returning one scalar. Annotate its return type with the requested CUTLASS dtype. The function must implement an associative operation for scan and reduction. Runtime captures and stateful prefix callbacks are unsupported.

  • dtype (type) – A supported CUTLASS or NumPy scalar type, or int / float.

  • arch (str or None) – CUDA target such as "sm_90a". None uses the configured CuTe target or the current GPU. Kernels using the result must target the same arch.

Returns:

Immutable compiled operator accepted by scan and reduction.

Return type:

DeviceOperator

class cuda.coop.cutlass.DeviceOperator(
dtype: type,
symbol: str,
cpp_type: str,
arch: str,
_ltoir: bytes,
)#

A stateless scalar binary operator prepared by compile_operator().

dtype is the CUTLASS type of both operands and the return value. arch records the CUDA target required by kernels using this operator. The compiled device code is retained so a removed cache file can be rebuilt.

Thread groups#

cuda.coop.cutlass.this_thread() → ThreadGroup#

Describe the calling CUDA thread.

cuda.coop.cutlass.this_warp() → ThreadGroup#

Describe the calling physical 32-thread warp.

Construction does not inspect active lanes or synchronize threads. A consuming query or primitive resolves the required launch facts and applies that operation’s participation rules.

cuda.coop.cutlass.this_block() → ThreadGroup#

Describe the current CUDA thread block for a CUTLASS primitive.

cuda.coop.cutlass.this_cluster() → ThreadGroup#

Describe the cluster of the current kernel launch.

The consuming query or primitive resolves launch facts. Constructing this descriptor does not configure cluster scheduling.

cuda.coop.cutlass.this_grid() → ThreadGroup#

Describe the current grid for supported hierarchy queries.

Queries resolve the launch dimensions when consumed. This descriptor does not enable grid synchronization or reductions.

class cuda.coop.cutlass.ThreadGroup(
kind: str,
hierarchy: ThreadHierarchy = <factory>,
parent: ThreadGroup | None = None,
mapping: GroupByMapping | None = None,
source: str = 'explicit',
)#

Describe a group and emit CuTe hierarchy queries on demand.

Construction and group_by create descriptors. Rank, count, membership, and synchronization methods emit device operations when called during tracing. Query results are CuTe scalar values. See thread groups for hierarchy and participation.

group_by(
count: int,
*,
exhaustive: bool = True,
) → _ThreadGroupT#

Partition a physical warp by threads or a block by warps.

Parameters:
  • count (int) – Compile-time positive number of units in each group. The units are threads for a physical-warp parent and physical warps for a block parent. For example, this_block().group_by(2) describes 64 threads. The count cannot exceed the parent’s unit count.

  • exhaustive (bool, optional) – Compile-time flag, default True. If true, count must divide the parent’s unit count exactly. If false, trailing units that cannot form a complete group are excluded; query is_member() before using their ranks.

Returns:

A descriptor for the calling thread’s mapped group. Nested group_by() calls are unsupported.

Return type:

cuda.coop.cutlass.ThreadGroup

Notes

The enclosing block must contain complete physical warps. Logical Warp primitives support widths of 1, 2, 4, 8, 16, or 32. Mapped groups may query their constituents and immediate physical parent, but not a higher hierarchy level. Groups of physical warps support queries and have no explicit synchronization methods. See thread groups and participation requirements.

rank(level='thread')#

Query rank with the default unsigned CuTe result type.

The optional level selects the queried hierarchy level. Queries involving the grid return Uint64; other queries return Uint32. See thread groups for the hierarchy.

count(level='thread')#

Query count with the default unsigned CuTe result type.

The optional level selects the queried hierarchy level. Queries involving the grid return Uint64; other queries return Uint32. See thread groups for the hierarchy.

rank_as(dtype=None, level='thread')#

Query rank using an optional integer dtype selector.

The result remains a CuTe scalar even when dtype is a Python or NumPy type. None keeps the default unsigned width. level has the same meaning as in cuda.coop.cutlass.ThreadGroup.rank().

count_as(dtype=None, level='thread')#

Query count using an optional integer dtype selector.

The result remains a CuTe scalar even when dtype is a Python or NumPy type. None keeps the default unsigned width. level has the same meaning as in cuda.coop.cutlass.ThreadGroup.count().

is_member()#

Return a CuTe Uint8 flag for membership in this group.

Check membership before using a mapped rank. Primitive participation rules still apply; a membership guard alone cannot make a divergent collective valid.

sync()#

Synchronize the participating members of this group.

sync_aligned()#

Synchronize a converged group with aligned participation.

class cuda.coop.cutlass.ThreadHierarchy#

CUDA hierarchy descriptor for the current kernel launch.

ThreadHierarchy() and ThreadHierarchy.current() describe the active launch. Hierarchy is an alias for this class. The compiler supplies the block, grid, and optional cluster dimensions from the kernel launch; this constructor accepts no dimensions.

Use cuda.coop.cutlass.this_block(), cuda.coop.cutlass.this_warp(), or another group factory in a kernel to obtain a cuda.coop.cutlass.ThreadGroup. See thread groups for the hierarchy and ranks and sizes for runtime queries. The corresponding C++ abstraction is cuda::hierarchy.

Notes

ThreadHierarchy() and current() leave block_dim, grid_dim, and cluster_dim as None. Creating one does not resolve runtime ranks or allocate device storage. Use group methods inside a kernel to query the actual launch.

class cuda.coop.cutlass.Hierarchy#

Alias for ThreadHierarchy.

Per-thread payloads and CuTe conversion#

class cuda.coop.cutlass.ThreadData(
items_per_thread: int,
dtype: Any = None,
*,
values: tuple[Any, ...] | list[Any] | None = None,
alignment: int | None = None,
)#

Hold a fixed number of items per thread inside a CuTe kernel.

Primitives infer the item count from the payload. Indexing uses compile-time integers; initialize each item before reading it or passing it to an operation that reads the payload.

The conversion methods below copy values between this payload and CuTe registers. Use cuda.coop.cutlass.load() and cuda.coop.cutlass.store() to move data to or from global or shared memory. See CUTLASS register payloads for the distinction between mutable register tensors and TensorSSA.

Parameters:
  • items_per_thread (int) – Positive compile-time item count.

  • dtype (type, optional) – Optional numeric element-type metadata. Load supplies the memory element type; consuming primitives can infer it from homogeneous initialized values. Advanced interop may need explicit metadata when a raw integer IR value has no signedness. See Element types in advanced interop. The constructor does not cast entries supplied in values.

  • values (tuple or list, optional) – Initial values, with exactly items_per_thread entries. When omitted, the items remain uninitialized until assigned or loaded.

  • alignment (int, optional) – Minimum byte alignment for materialized storage. Must be a positive power of two. The backend may select a larger alignment.

Examples

Convert a CuTe register tensor to a payload, change one item, and export both immutable and mutable register results. The original tensor retains its values:

import cutlass
from cutlass import cute

import cuda.coop.cutlass as cutlass_coop

@cute.kernel
def convert_registers(destination: cute.Pointer):
    thread = cute.arch.thread_idx()[0]
    original = cute.make_rmem_tensor(2, cutlass.Int32)
    original[0] = thread * 2
    original[1] = thread * 2 + 1

    items = cutlass_coop.ThreadData.from_register_tensor(original)
    items[0] = items[0] + 10
    vector = items.to_tensor_ssa()
    copied = items.to_register_tensor()
    copied[1] = copied[1] + 100

    output = cute.make_tensor(destination, cute.make_layout(64 * 6))
    for item in cutlass.range_constexpr(2):
        output[thread * 6 + item] = original[item]
        output[thread * 6 + 2 + item] = vector[item]
        output[thread * 6 + 4 + item] = copied[item]

@cute.jit
def launch(destination: cute.Pointer):
    convert_registers(destination).launch(grid=1, block=64)

classmethod ThreadData.from_values(
*values: Any,
dtype: Any = None,
) → ThreadData#

Construct a payload from one or more per-thread scalar values.

Parameters:
  • *values (scalar) – Initial items, in payload order. At least one value is required.

  • dtype (type, optional) – Dtype metadata. Values are stored as supplied, without casting.

Returns:

A new payload containing len(values) initialized items.

Return type:

ThreadData

classmethod ThreadData.from_fn(
items_per_thread: int,
fn: Callable[[int], Any],
*,
dtype: Any = None,
) → ThreadData#

Initialize each item with a compile-time Python callable.

Parameters:
  • items_per_thread (int) – Positive compile-time item count.

  • fn (callable) – Called once for each integer index from zero to items_per_thread - 1 while tracing the kernel. It may build CuTe scalar expressions; it is not a device callback.

  • dtype (type, optional) – Dtype for the generated values. An explicit CUTLASS numeric type casts each item to that type.

Returns:

A new initialized payload in increasing item-index order.

Return type:

ThreadData

classmethod ThreadData.from_register_tensor(
fragment: Any,
*,
items_per_thread: int | None = None,
dtype: Any = None,
) → ThreadData#

Copy a CuTe register tensor into a per-thread payload.

Parameters:
  • fragment (cute.Tensor) – Tensor in CuTe register memory (rmem). Global- and shared-memory tensors are rejected; load their data first.

  • items_per_thread (int, optional) – Positive compile-time item count. Inferred from the static tensor shape when available. An explicit count must match the inferred count; it is required when inference is unavailable.

  • dtype (type, optional) – Defaults to fragment.element_type. An explicit CUTLASS numeric type casts each extracted item.

Returns:

A new payload in the tensor’s flat integer-index order. Subsequent assignments to either object do not update the other.

Return type:

ThreadData

classmethod ThreadData.from_vector(
vector: Any,
*,
items_per_thread: int | None = None,
dtype: Any = None,
) → ThreadData#

Copy an immutable register value into a per-thread payload.

Parameters:
  • vector (cute.TensorSSA or CUTLASS register vector) – Integer-indexable register value. Memory-backed tensors and arrays are rejected. For mutable CuTe rmem tensors, use from_register_tensor().

  • items_per_thread (int, optional) – Positive compile-time item count, normally inferred from numel() or a static one-dimensional shape. An explicit count must match any inferred count.

  • dtype (type, optional) – Defaults to the value’s dtype metadata. An explicit CUTLASS numeric type casts each extracted item.

Returns:

A new payload in flat integer-index order. The input is unchanged.

Return type:

ThreadData

classmethod ThreadData.from_payload(
payload: Any,
*,
items_per_thread: int | None = None,
dtype: Any = None,
) → ThreadData#

Adapt a ThreadData or CuTe register payload.

Parameters:
  • payload (ThreadData, cute.Tensor, cute.TensorSSA, or register vector) – Per-thread values. CuTe tensors must use register memory; global/shared tensors and arrays require a load first.

  • items_per_thread (int, optional) – Positive compile-time item count. If supplied, it must match the existing or inferred count.

  • dtype (type, optional) – Dtype for conversion. For an existing ThreadData with dtype metadata, this must match that dtype; this method does not recast an already typed payload.

Returns:

The same object when payload is already ThreadData and no dtype change is needed. Supplying a dtype to an untyped ThreadData creates a new payload. CuTe inputs produce a new payload through from_register_tensor() or from_vector().

Return type:

ThreadData

classmethod ThreadData.load(
source: ThreadDataLoadSource,
*,
items_per_thread: int | None = None,
dtype: Any = None,
) → ThreadData#

Load a producer-owned source into per-thread registers.

The source must explicitly implement the private __cuda_coop_thread_data_load__ tracing hook. The producer owns any required tensor-memory-to-register copy, including its copy atom, thread partition, readiness, and lifetime. The hook must return either ThreadData or a statically sized per-thread register payload accepted by from_payload(). The hook is invoked exactly once by each call during DSL tracing; a single-use producer source is responsible for rejecting a second call.

Bare memory-backed tensors are intentionally not interpreted here: their address space and shape do not provide enough information to safely select or schedule a register load.

items_per_thread and dtype are optional static constraints on the returned register payload, matching from_payload().

ThreadData.to_tensor_ssa(
*,
dtype: Any = None,
shape: Any = None,
like: ThreadDataTensorMetadata | None = None,
) → Any#

Export initialized items as an immutable CuTe register value.

Parameters:
  • dtype (type, optional) – Supported CUTLASS or NumPy numeric dtype. Defaults to this payload’s dtype; required if the payload has no dtype metadata.

  • shape (int or tuple, optional) – Defaults to (items_per_thread,). May be nested, but every extent must be a positive compile-time integer and the total number of elements must equal items_per_thread.

  • like (ThreadDataTensorMetadata, optional) – Supplies missing shape and dtype metadata. Explicit arguments take precedence. Flat item order is preserved; producer-specific lane ownership is not reconstructed by this metadata.

Returns:

Register values with the requested dtype and shape, in payload item order. The original input fragment’s shape is not retained automatically. Later payload assignments do not change this value.

Return type:

cute.TensorSSA

ThreadData.to_register_tensor(
*,
dtype: Any = None,
shape: Any = None,
) → Any#

Export initialized items to a new mutable CuTe register tensor.

Parameters:
Returns:

A new rmem tensor containing the payload values. It does not alias the payload or an earlier input fragment. Its storage honors this payload’s minimum alignment. As with other register tensors, the compiler may spill values to local memory.

Return type:

cute.Tensor

Producer protocols#

class cuda.coop.cutlass.ThreadDataLoadSource(*args, **kwargs)#

Producer capability that loads one payload into per-thread registers.

Implementations own the storage-specific copy policy and expose it through this trace-time hook.

class cuda.coop.cutlass.ThreadDataTensorMetadata(*args, **kwargs)#

Static shape and dtype used to reconstruct a CuTe TensorSSA.

class cuda.coop.cutlass.ThreadDataSource(*args, **kwargs)#

Producer source that supports loading and TensorSSA reconstruction.

See producer interoperability for capability detection, copy ownership, and lifetime requirements.

Temporary storage#

class cuda.coop.cutlass.TempStorage(
size_in_bytes=None,
*,
alignment=None,
auto_sync=False,
sharing='shared',
)#

Describe shared-memory requirements for operations in one kernel.

Supported block algorithms accept an explicit descriptor. reserve exposes typed arrays for application and library code. See temporary storage for allocation lifetime and the CUTLASS example for reuse.

size_in_bytes#

Explicit positive capacity, or None to infer it from all uses, including reserved arrays. An explicit capacity must cover the resolved allocation and fit a signed 32-bit integer. Device shared-memory limits still apply.

Type:

int or None

alignment#

Optional minimum byte alignment, a power of two up to 1024. The planner also satisfies the C++ primitive’s alignment requirement.

Type:

int or None

auto_sync#

Whether each scratch-using call emits a trailing block barrier. Defaults to False; None is normalized to False. Without automatic barriers, the caller must synchronize before scratch reuse.

Type:

bool

sharing#

Shared calls and reservations alias one slice sized for the largest requirement. Exclusive calls and reservations get separate slices in trace order. Repeated execution of one call site reuses its slice, including in a loop.

Type:

{“shared”, “exclusive”}

Equal settings do not combine separate descriptors. Reuse the same object
to share an allocation across calls in a kernel. Finalization validates
descriptor lifetimes and supplies the storage operands. ``release``
consumes the descriptor and returns its complete region as a native view.
Runtime branches and loops may carry aliases of the same kernel-local
descriptor; selecting or rebinding to a different descriptor is rejected.
Use an ``if`` statement for conditional operations, with a condition uniform
across participating threads, rather than a runtime conditional expression.
TempStorage.reserve(num_elems, dtype, *, alignment=None)#

Return a one-dimensional CuTe shared tensor with a fixed extent.

num_elems, dtype, and optional byte alignment must be compile-time constants. Supported dtypes are 8/16/32/64-bit signed and unsigned integers, Float16, BFloat16, Float32, and Float64. Ordinary NumPy scalar types and dtypes are accepted where available.

The descriptor’s sharing policy applies to both reservations and primitive scratch. Shared views alias the same bytes; exclusive views occupy separate slices. Each traced call site has one view, including when it executes repeatedly in a runtime loop. Separate descriptors always own separate allocations.

Reservations require auto_sync=False. The caller must finish all accesses and asynchronous library operations before reusing overlapping bytes, and synchronize participating threads as needed. sync() is a block barrier, not an asynchronous completion wait.

TempStorage.release(dtype=None)#

Return the entire existing region as a native shared CuTe tensor.

dtype is a supported compile-time scalar type; omission returns bytes (Uint8). The resolved capacity includes exclusive slices, padding, and any explicit capacity. Its size must be divisible by the element size and its alignment must satisfy that type. An empty inferred region is rejected. Inferred extents resolve at finalization. The descriptor must be constructed inside the kernel.

This consumes every alias of the descriptor on the executing path. Subsequent primitives, reserve, sync, or release calls are compile errors, including across loop iterations. A fresh constructor inside a loop creates a new logical instance. Release adds no allocation, barrier, or asynchronous completion wait; the caller completes prior work before using overlapping bytes through the returned tensor.

Memory operations#

cuda.coop.cutlass.load(
group: ThreadGroup,
source: object,
/,
*,
items_per_thread: int,
dtype: object | None = None,
alignment: int | None = None,
algorithm: Literal['direct', 'striped', 'vectorize', 'transpose', 'warp_transpose', 'warp_transpose_timesliced'] = 'direct',
valid_items: int | integer | CompilerIntegerLike | None = None,
oob_default: _NumericT | float | None = None,
offset: int | integer | CompilerIntegerLike | None = None,
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Load a contiguous group tile into per-thread values.

Supply a positive compile-time items_per_thread to return fresh ThreadData with the source dtype. The result owns independent values, including when the count is one. Blocks and physical or logical warps are supported.

The source dtype is the element type its CuTe operand reports. CuTe reports integer memory behind a cute.Pointer as signed, including when the host created the pointer with an unsigned type. The exception is a pointer that the kernel itself made with cute.make_ptr and passes unchanged. A cute.make_tensor view of such memory is signed as well. Uint32 data then loads as Int32 with no error, and a maximum, a sort, or a widening conversion gives the signed answer.

For unsigned memory, pass dtype with the unsigned type of the same width, such as dtype=cutlass.Uint32. The result has that dtype, and oob_default is checked against it. dtype may also repeat the reported type. Any other dtype is an error, so it cannot change the element width or turn integers into floats. A cute.recast_tensor view states its element type and needs no dtype.

alignment requests a minimum power-of-two byte alignment for the result’s materialized storage, as in ThreadData. It does not describe source memory alignment or change the selected item layout. To update a native CuTe register tensor, use tensor.store(values.to_tensor_ssa()) with compatible shape and item order. The Load implementation materializes a temporary register tensor and copies its scalar values into ThreadData.

Load populates the payload in the selected algorithm’s layout. Beyond valid_items, slots have unspecified values unless oob_default is supplied. A default requires valid_items. A runtime default must have the memory dtype. A Python or NumPy default must be finite and within the memory dtype’s range; a float default requires a floating-point dtype.

The count ranges from zero through the group’s full tile size. offset is a nonnegative element offset. Counts, offsets, and supplied defaults must agree within the group. Physical and logical warps receive consecutive tiles in linear block-thread order; the compiler adds that group’s tile origin to the offset. Different groups can use different controls. The caller must provide enough accessible memory for the selected prefix at the resulting offset.

For this_warp().group_by(width), width must be 1, 2, 4, 8, 16, or 32. Each logical tile contains width * items_per_thread elements. The enclosing block must contain complete 32-thread physical warps.

The source must expose a raw pointer and a provably compact layout. A bare pointer object with no shape or stride metadata is also accepted, but its capacity is not checked. Register or local-memory tensors are rejected. Load reads addressable memory, such as global or shared memory. Shared tensors from SmemAllocator and TempStorage.reserve are supported when their dtype and compact layout meet these requirements. Synchronize the participating threads after shared-memory writes and before Load reads values produced by other threads. DIRECT, STRIPED, and VECTORIZE need no shared scratch or reuse barrier; an accepted block descriptor does not change that.

Transpose algorithms use shared scratch. With no temp_storage, the compiler allocates it and inserts a trailing reuse barrier. An explicit cuda.coop.cutlass.TempStorage is supported only for block calls. It sets size, alignment, sharing, and synchronization policy; its default auto_sync=False requires a barrier before reuse. Each physical or logical warp uses an independent scratch slice and a reuse barrier that covers only its own lanes.

Examples

Copy a partial tile between different source and destination offsets. The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def copy_prefix(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    valid_items = 64 * items_per_thread - 3
    values = coop.load(
        block,
        source,
        items_per_thread=items_per_thread,
        valid_items=valid_items,
        oob_default=0,
        offset=5,
    )
    coop.store(
        block,
        destination,
        values,
        valid_items=valid_items,
        offset=2,
    )

@cute.jit
def launch(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    copy_prefix(source, destination, items_per_thread).launch(
        grid=1, block=64
    )

cuda.coop.cutlass.store(
group: ThreadGroup,
destination: object,
value: _NumericT | _ReadableThreadDataLike[_NumericT],
/,
*,
dtype: object | None = None,
algorithm: Literal['direct', 'striped', 'vectorize', 'transpose', 'warp_transpose', 'warp_transpose_timesliced'] = 'direct',
valid_items: int | integer | CompilerIntegerLike | None = None,
offset: int | integer | CompilerIntegerLike | None = None,
temp_storage: TempStorageLike | None = None,
) → None#

Store per-thread values into a contiguous group tile.

Blocks and physical or logical warps are supported. Each thread supplies a scalar or an initialized CUTLASS ThreadData payload whose dtype matches the destination. Do not rely on the payload’s contents after a transpose Store; copy values needed later.

dtype declares the destination’s element type as it does for load(). To store an unsigned payload through a cute.Pointer, which reports the signed type of the same width, pass the payload’s dtype. The declared type must equal the payload dtype.

valid_items selects a prefix from zero through the group’s full tile size. offset is a nonnegative element offset. Both must agree within the group. Physical and logical warp widths, tile origins, and prefix bounds follow load(). Different groups can use different controls. The caller must provide enough accessible destination memory for the prefix at the resulting offset. Items outside it are not written. The destination has the same pointer and layout requirements as load().

Transpose algorithms use shared scratch. With no temp_storage, the compiler allocates it and inserts a trailing reuse barrier. An explicit descriptor is supported only for block calls and controls allocation and reuse. The caller must synchronize before reuse unless auto_sync=True. Physical and logical warps use independent scratch slices and a barrier that covers only the group’s lanes. DIRECT, STRIPED, and VECTORIZE need no scratch or reuse barrier.

Examples

Copy a partial tile between different source and destination offsets. The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def copy_prefix(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    valid_items = 64 * items_per_thread - 3
    values = coop.load(
        block,
        source,
        items_per_thread=items_per_thread,
        valid_items=valid_items,
        oob_default=0,
        offset=5,
    )
    coop.store(
        block,
        destination,
        values,
        valid_items=valid_items,
        offset=2,
    )

@cute.jit
def launch(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    copy_prefix(source, destination, items_per_thread).launch(
        grid=1, block=64
    )

Reduction#

cuda.coop.cutlass.reduce(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | _ItemT | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
binary_op: Literal['+', 'sum', 'add', 'plus', '*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
valid_items: int | integer | CompilerIntegerLike | None = None,
algorithm: Literal['raking_commutative_only', 'raking', 'warp_reductions'] | None = None,
temp_storage: TempStorageLike | None = None,
) → _ItemT#

Reduce scalars or register payloads with an associative operator.

Every member of the block or warp participates. Only group rank zero may read the result. CuTe register payloads contribute all their elements.

Parameters:
  • group (ThreadGroup) – Block, physical warp, or logical warp. Logical widths may be powers of two from 1 through 32, or widths from 17 through 31. For the latter, use group_by(width, exhaustive=False) and guard the call with group.is_member(). Every member participates, including members excluded by valid_items. The enclosing block must contain complete physical warps for warp reductions.

  • value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Each member supplies one scalar or a fixed-size per-thread payload. Register tensors and TensorSSA values are converted with cuda.coop.cutlass.ThreadData.from_payload(). Every payload element contributes to one scalar result. Inputs remain unchanged. Supported dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64.

  • binary_op (str, built-in alias, or DeviceOperator, optional) – Compile-time operator, default sum. Supported strings are "sum", "multiplies", "min", "max", "bit_and", "bit_or", and "bit_xor". Also accepts the corresponding operator or NumPy aliases, such as operator.add and numpy.maximum. Bitwise operators require an integer dtype. A device operator prepared with compile_operator() must accept and return the input dtype and be associative. It cannot use "raking_commutative_only".

  • valid_items (integer, optional) – Group-uniform count of contributing members, from one through the group size. Requires scalar input. None includes every member; empty reductions are unsupported.

  • algorithm (str, optional) – Compile-time block algorithm: "raking_commutative_only", "raking", or "warp_reductions". None selects "warp_reductions". Warp groups require None.

  • temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Size and alignment are inferred from its uses unless requested. Omission enables automatic allocation and reuse barriers. With auto_sync=False, synchronize the block before reuse. Warp groups use automatic storage and require None.

Returns:

Reduced value with the input dtype, defined only at group rank zero. A NumPy dtype selector still produces a CuTe scalar inside the kernel.

Return type:

CuTe numeric scalar

Notes

Every reduction uses CUB. Blocks and warps accept scalars or per-thread payloads. A valid_items prefix counts contributing members, requires scalar input, and does not reduce required participation. Block reductions accept temp_storage with the same sharing, capacity, alignment, and synchronization policies as Load/Store. Floating-point results can differ from a sequential fold because reduction regroups operations.

See also

cuda.coop.cutlass.sum

Sum with the same operand and result behavior.

cub::BlockReduce, cub::WarpReduce

C++ primitives used for block and warp reductions.

Examples

Compute a sum and a maximum over the same tile. Only the block root reads either result.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def summarize(
    source: cute.Pointer,
    totals: cute.Pointer,
    maxima: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    total = coop.sum(block, values)
    maximum = coop.reduce(block, values, binary_op="max")
    total_output = cute.make_tensor(totals, cute.make_layout(1))
    maximum_output = cute.make_tensor(maxima, cute.make_layout(1))
    if block.rank() == 0:
        total_output[0] = total
        maximum_output[0] = maximum

@cute.jit
def launch(
    source: cute.Pointer,
    totals: cute.Pointer,
    maxima: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    summarize(source, totals, maxima, items_per_thread).launch(
        grid=1, block=64
    )

cuda.coop.cutlass.reduce_batched(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
binary_op: Literal['+', 'sum', 'add', 'plus', '*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | Callable[[object, object], object] | None = None,
output_layout: Literal['striped', 'blocked'] = 'striped',
) → ThreadData#

Reduce each payload slot independently across a warp.

Each slot is an independent batch: slot b in every lane contributes to batch b. The input is preserved. Readable ThreadData, CuTe register tensors, and TensorSSA values are accepted through ThreadData.from_payload.

binary_op accepts the built-in strings and aliases documented by cuda.coop.cutlass.reduce(). Custom callbacks are not supported. output_layout is "striped" or "blocked". For B batches and W lanes, the returned ThreadData contains ceil(B / W) slots per lane. Only slots corresponding to a batch are defined. Guard reads using the batch index.

The CUB provider exchanges register values within the participating warp. It requires no shared scratch allocation or storage-reuse barrier. Logical warps may call it independently, including from different branches.

Parameters:
  • group (ThreadGroup) – Complete physical warp or logical warp of 1, 2, 4, 8, 16, or 32 lanes. All members of the selected group participate.

  • value (ThreadData or CuTe register payload) – One input per independent batch in each lane. All lanes use the same positive batch count and dtype: signed or unsigned 8-, 16-, 32-, or 64-bit integers, Float32, or Float64. The input is preserved.

  • binary_op (str or recognized built-in alias, optional) – Reduction operator, default addition. Bitwise operators require integer inputs. The operator must be associative and commutative. Custom callbacks are not supported.

  • output_layout ({"striped", "blocked"}, optional) – Result ownership, default striped. Striped slot i in lane t holds batch t + i * W; blocked slot i holds batch t * ceil(B / W) + i. Read only indices below B.

Returns:

Fresh payload with the input dtype and ceil(B / W) slots per lane. Accumulation uses the input dtype without promotion. Slots beyond the batch count are undefined.

Return type:

ThreadData

Notes

See the Batched Warp Reduction visualization for lane and result ownership.

Examples

Sum each per-thread slot independently within two physical warps. Guard result reads by the batch index: lanes without a batch have no defined result.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def sum_columns(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    warp = coop.this_warp()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    totals = coop.reduce_batched(warp, values)
    output = cute.make_tensor(
        destination, cute.make_layout(2 * items_per_thread)
    )
    for item in cutlass.range_constexpr((items_per_thread + 31) // 32):
        batch = warp.rank() + item * 32
        if batch < items_per_thread:
            index = block.rank() // 32 * items_per_thread + batch
            output[index] = totals[item]

@cute.jit
def launch(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    sum_columns(source, destination, items_per_thread).launch(
        grid=1, block=64
    )

cuda.coop.cutlass.sum(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | _ItemT | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
valid_items: int | integer | CompilerIntegerLike | None = None,
algorithm: Literal['raking_commutative_only', 'raking', 'warp_reductions'] | None = None,
temp_storage: TempStorageLike | None = None,
) → _ItemT#

Sum scalars or per-thread register payloads with CUTLASS.

Equivalent to cuda.coop.cutlass.reduce() with binary_op="sum". Its group, payload, valid-prefix, algorithm, and scratch contracts apply. Every group member participates; only rank zero may read the result.

Parameters:
  • group (ThreadGroup) – Block, physical warp, or logical warp supported by reduce.

  • value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Each member’s contribution. Every payload item contributes; the input remains unchanged.

  • valid_items (integer, optional) – Uniform count of contributing members, for scalar inputs only.

  • algorithm (str, optional) – Block algorithm described by reduce; omit for warp groups.

  • temp_storage (TempStorage, optional) – Explicit block scratch descriptor; omit for warp groups. Omission enables automatic scratch and reuse synchronization.

Returns:

Sum with the input dtype, defined only at group rank zero.

Return type:

CuTe numeric scalar

See also

cuda.coop.cutlass.reduce

Built-in operators, register payloads, and result ownership.

cub::BlockReduce, cub::WarpReduce

C++ primitives used for block and warp reductions.

Examples

Write a tile sum and maximum from the block root.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def summarize(
    source: cute.Pointer,
    totals: cute.Pointer,
    maxima: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    total = coop.sum(block, values)
    maximum = coop.reduce(block, values, binary_op="max")
    total_output = cute.make_tensor(totals, cute.make_layout(1))
    maximum_output = cute.make_tensor(maxima, cute.make_layout(1))
    if block.rank() == 0:
        total_output[0] = total
        maximum_output[0] = maximum

@cute.jit
def launch(
    source: cute.Pointer,
    totals: cute.Pointer,
    maxima: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    summarize(source, totals, maxima, items_per_thread).launch(
        grid=1, block=64
    )

Scan#

cuda.coop.cutlass.scan(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
/,
*,
mode: Literal['exclusive', 'inclusive'] = 'exclusive',
scan_op: Literal['+', 'sum', 'add', 'plus'] | Literal['*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
initial_value: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
temp_storage: TempStorageLike | None = None,
valid_items: int | integer | CompilerIntegerLike | None = None,
aggregate_output: ThreadDataLike[Any] | None = None,
) → ThreadData | _ItemT#

Scan register values with optional valid-prefix and aggregate controls.

Prefixes follow linear group rank. Block payloads use blocked order: all items from thread zero, then all items from thread one, and so on. Prefixes restart for each group; this call does not scan across blocks.

Parameters:
  • group (ThreadGroup) – Complete block, physical warp, or supported logical warp. Every group member executes the same call. Warp scans require an enclosing block size divisible by 32.

  • value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Blocks accept scalars or fixed-size per-thread payloads in blocked order. Register tensors and TensorSSA values are converted with cuda.coop.cutlass.ThreadData.from_payload(). All members must use the same dtype and extent. Physical and logical warps accept scalars only, including when each thread would otherwise hold one item.

  • mode ({"exclusive", "inclusive"}, optional) – Compile-time choice, default "exclusive". Exclusive prefixes combine initial_value with preceding inputs. Inclusive prefixes also include the current item and require no initial value.

  • scan_op (str, built-in alias, or DeviceOperator, optional) – Compile-time operator, default sum. Supports "sum", "multiplies", "min", "max", "bit_and", "bit_or", and "bit_xor", and corresponding operator or NumPy aliases such as operator.add and numpy.maximum. Bitwise operators require an integer dtype. A device operator prepared with compile_operator() must accept and return the input dtype and be associative. Exclusive mode requires an explicit initial value.

  • initial_value (numeric scalar, optional) – Starting value for exclusive mode. Sum defaults to zero; other operators require an explicit value. A typed CuTe or NumPy scalar must match the input dtype. Python literals must be finite and representable in that dtype. Inclusive mode requires None. The initial value must be uniform across the group.

  • valid_items (int or CuTe integer scalar, optional) – Warp-only count of contributing lanes, from one through the group size, uniform within the group. None includes every lane. All lanes participate, but only ranks below this count have defined scan results. The enclosing block must contain complete physical warps. Blocks require None.

  • aggregate_output (ThreadData, optional) – Writable one-item payload receiving the input aggregate on every member. Its dtype must match the input, or may be omitted for inference. The aggregate excludes initial_value and lanes beyond valid_items. CuTe register tensors and TensorSSA values are not accepted as this output argument. None omits the aggregate. Inside a runtime branch or loop, create this output in that region and explicitly copy its item to carry the value beyond the region.

  • algorithm (str, optional) – Compile-time block algorithm: "raking" (the default), "raking_memoize", or "warp_scans". The last requires a block size divisible by 32. Warp groups require None.

  • temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Omission enables automatic scratch and reuse synchronization. Warp groups require None and use automatic storage for each group.

Returns:

This thread’s prefixes with the input dtype. Scalar input returns a CuTe scalar; payload input returns a new writable ThreadData with the input extent and alignment. The input remains unchanged. With valid_items, read only valid lanes. aggregate_output is written separately, including on lanes outside the valid prefix.

Return type:

CuTe numeric scalar or cuda.coop.cutlass.ThreadData

Notes

Only blocks accept algorithm and explicit temp_storage. The compiler synchronizes automatically when it manages scratch. An explicit cuda.coop.cutlass.TempStorage defaults to auto_sync=False. It requires block barriers before reuse; request auto_sync=True for automatic reuse synchronization. Prefix callbacks and callback state are not supported. Scan requires an associative operator and may regroup arithmetic. Floating-point prefixes can differ from a sequential fold; integer arithmetic uses the input dtype.

See also

cuda.coop.cutlass.exclusive_scan

Executable example of a partial logical warp and aggregate output.

cub::BlockScan, cub::WarpScan

C++ scan, sum, and aggregate overloads.

Examples

Compare seeded exclusive sums, inclusive maxima, and the sum helpers. One automatically synchronized scratch descriptor serves all four calls.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def prefixes(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    scratch = coop.TempStorage(auto_sync=True)
    seeded_sums = coop.scan(
        block, values, initial_value=7, temp_storage=scratch
    )
    running_maxima = coop.inclusive_scan(
        block, values, scan_op="max", temp_storage=scratch
    )
    before = coop.exclusive_sum(block, values, temp_storage=scratch)
    through = coop.inclusive_sum(block, values, temp_storage=scratch)
    coop.store(block, seeded, seeded_sums)
    coop.store(block, maxima, running_maxima)
    coop.store(block, exclusive, before)
    coop.store(block, inclusive, through)

@cute.jit
def launch(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    prefixes(
        source, seeded, maxima, exclusive, inclusive, items_per_thread
    ).launch(grid=1, block=64)

cuda.coop.cutlass.exclusive_sum(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
/,
*,
algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
temp_storage: TempStorageLike | None = None,
valid_items: int | integer | CompilerIntegerLike | None = None,
aggregate_output: ThreadDataLike[Any] | None = None,
) → ThreadData | _ItemT#

Return exclusive sums starting from zero.

Uses the parameters and return behavior of cuda.coop.cutlass.scan(), with exclusive mode and sum fixed. Use cuda.coop.cutlass.exclusive_scan() for an explicit initial_value or a different associative operator. The valid_items and aggregate_output controls are available here too.

Returns:

Sum of the items preceding each input item, with the input dtype. Payload input produces a fresh payload and remains unchanged. Read only valid lanes when valid_items is supplied.

Return type:

CuTe numeric scalar or cuda.coop.cutlass.ThreadData

See also

cuda.coop.cutlass.exclusive_scan

Partial-warp example. For an exclusive sum starting at zero, replace the call with exclusive_sum and omit scan_op and initial_value.

Examples

Compute prefixes before each item and compare them with inclusive sums and an exclusive scan starting from seven.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def prefixes(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    scratch = coop.TempStorage(auto_sync=True)
    seeded_sums = coop.scan(
        block, values, initial_value=7, temp_storage=scratch
    )
    running_maxima = coop.inclusive_scan(
        block, values, scan_op="max", temp_storage=scratch
    )
    before = coop.exclusive_sum(block, values, temp_storage=scratch)
    through = coop.inclusive_sum(block, values, temp_storage=scratch)
    coop.store(block, seeded, seeded_sums)
    coop.store(block, maxima, running_maxima)
    coop.store(block, exclusive, before)
    coop.store(block, inclusive, through)

@cute.jit
def launch(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    prefixes(
        source, seeded, maxima, exclusive, inclusive, items_per_thread
    ).launch(grid=1, block=64)

cuda.coop.cutlass.inclusive_sum(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
/,
*,
algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
temp_storage: TempStorageLike | None = None,
valid_items: int | integer | CompilerIntegerLike | None = None,
aggregate_output: ThreadDataLike[Any] | None = None,
) → ThreadData | _ItemT#

Return inclusive sums, including each current item.

Uses the parameters and return behavior of cuda.coop.cutlass.scan(), with inclusive mode and sum fixed. The valid_items and aggregate_output controls follow that function.

Returns:

Sum through each input item, with the input dtype. Payload input produces a fresh payload and remains unchanged. Read only valid lanes when valid_items is supplied.

Return type:

CuTe numeric scalar or cuda.coop.cutlass.ThreadData

See also

cuda.coop.cutlass.inclusive_scan

Inclusive prefixes with other associative operators.

cuda.coop.cutlass.exclusive_scan

Partial-warp example. For an inclusive sum, replace the call with inclusive_sum and omit scan_op and initial_value.

Examples

Compute prefixes through each item alongside exclusive sums and running maxima.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def prefixes(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    scratch = coop.TempStorage(auto_sync=True)
    seeded_sums = coop.scan(
        block, values, initial_value=7, temp_storage=scratch
    )
    running_maxima = coop.inclusive_scan(
        block, values, scan_op="max", temp_storage=scratch
    )
    before = coop.exclusive_sum(block, values, temp_storage=scratch)
    through = coop.inclusive_sum(block, values, temp_storage=scratch)
    coop.store(block, seeded, seeded_sums)
    coop.store(block, maxima, running_maxima)
    coop.store(block, exclusive, before)
    coop.store(block, inclusive, through)

@cute.jit
def launch(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    prefixes(
        source, seeded, maxima, exclusive, inclusive, items_per_thread
    ).launch(grid=1, block=64)

cuda.coop.cutlass.exclusive_scan(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
/,
*,
scan_op: Literal['+', 'sum', 'add', 'plus'] | Literal['*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
initial_value: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
temp_storage: TempStorageLike | None = None,
valid_items: int | integer | CompilerIntegerLike | None = None,
aggregate_output: ThreadDataLike[Any] | None = None,
) → ThreadData | _ItemT#

Return exclusive prefixes with an optional initial value and aggregate.

Uses the parameters and return behavior of cuda.coop.cutlass.scan(), with exclusive mode fixed. Sum starts from zero unless initial_value is supplied; other operators require it. aggregate_output receives the input aggregate without that initial value.

Returns:

Prefix before each input item, with the input dtype. Payload input produces a fresh payload. For a partial warp, only ranks below valid_items have defined prefixes.

Return type:

CuTe numeric scalar or cuda.coop.cutlass.ThreadData

Examples

Scan five lanes of each eight-lane logical warp, starting from seven. Every lane participates and receives the aggregate. Only valid lanes write their prefixes. operator.add selects the built-in sum.

import operator

from cutlass import cute

import cuda.coop.cutlass as cutlass_coop

@cute.kernel
def scan_prefix(
    source: cute.Pointer,
    destination: cute.Pointer,
    aggregates: cute.Pointer,
):
    thread = cutlass_coop.this_block().rank()
    group = cutlass_coop.this_warp().group_by(8)
    inputs = cute.make_tensor(source, cute.make_layout(64))
    outputs = cute.make_tensor(destination, cute.make_layout(64))
    totals = cute.make_tensor(aggregates, cute.make_layout(64))
    aggregate = cutlass_coop.ThreadData(items_per_thread=1)
    result = cutlass_coop.exclusive_scan(
        group,
        inputs[thread],
        scan_op=operator.add,
        initial_value=7,
        valid_items=5,
        aggregate_output=aggregate,
    )
    if group.rank() < 5:
        outputs[thread] = result
    totals[thread] = aggregate[0]

@cute.jit
def launch(
    source: cute.Pointer,
    destination: cute.Pointer,
    aggregates: cute.Pointer,
):
    scan_prefix(source, destination, aggregates).launch(grid=1, block=64)

cuda.coop.cutlass.inclusive_scan(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
/,
*,
scan_op: Literal['+', 'sum', 'add', 'plus'] | Literal['*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
temp_storage: TempStorageLike | None = None,
valid_items: int | integer | CompilerIntegerLike | None = None,
aggregate_output: ThreadDataLike[Any] | None = None,
) → ThreadData | _ItemT#

Return inclusive prefixes with an associative operator.

Uses the parameters and return behavior of cuda.coop.cutlass.scan(), with inclusive mode fixed. Each prefix includes its current item. This function has no initial_value parameter. The valid_items and aggregate_output controls follow cuda.coop.cutlass.scan().

Returns:

Inclusive prefixes with the input dtype. Payload input produces a fresh payload and remains unchanged. Read only valid lanes when valid_items is supplied.

Return type:

CuTe numeric scalar or cuda.coop.cutlass.ThreadData

See also

cuda.coop.cutlass.exclusive_scan

Partial-warp example. To include each current item, replace the call with inclusive_scan and remove initial_value.

Examples

Compute running maxima alongside seeded and unseeded prefix sums.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def prefixes(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    scratch = coop.TempStorage(auto_sync=True)
    seeded_sums = coop.scan(
        block, values, initial_value=7, temp_storage=scratch
    )
    running_maxima = coop.inclusive_scan(
        block, values, scan_op="max", temp_storage=scratch
    )
    before = coop.exclusive_sum(block, values, temp_storage=scratch)
    through = coop.inclusive_sum(block, values, temp_storage=scratch)
    coop.store(block, seeded, seeded_sums)
    coop.store(block, maxima, running_maxima)
    coop.store(block, exclusive, before)
    coop.store(block, inclusive, through)

@cute.jit
def launch(
    source: cute.Pointer,
    seeded: cute.Pointer,
    maxima: cute.Pointer,
    exclusive: cute.Pointer,
    inclusive: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    prefixes(
        source, seeded, maxima, exclusive, inclusive, items_per_thread
    ).launch(grid=1, block=64)

Data rearrangement#

cuda.coop.cutlass.exchange(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
mode: Literal['striped_to_blocked', 'blocked_to_striped', 'warp_striped_to_blocked', 'blocked_to_warp_striped', 'scatter_to_blocked', 'scatter_to_striped', 'scatter_to_striped_guarded', 'scatter_to_striped_flagged'] = 'striped_to_blocked',
ranks: _ReadableThreadDataLike[int | signedinteger[Any] | CompilerIntegerLike] | CutlassTensorSample | CutlassTensorSSASample | None = None,
valid_flags: _ReadableThreadDataLike[int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any]] | CutlassTensorSample | CutlassTensorSSASample | None = None,
warp_time_slicing: bool = False,
) → ThreadData#

Exchange register payloads, including ranked block scatters.

Every group member must call, including members whose scatter items are all invalid. For G threads and K items per thread, blocked order assigns logical element rank * K + item to a slot; striped order assigns item * G + rank. Exchange rearranges registers without loading or storing a memory tile.

Parameters:
  • group (ThreadGroup) – Complete block, physical warp, or logical warp with a power-of-two width from 1 through 32. Warp operations require an enclosing block size divisible by 32.

  • value (ThreadData, CuTe register tensor, or TensorSSA) – Fixed-size per-thread payload. Register tensors and TensorSSA values are converted with cuda.coop.cutlass.ThreadData.from_payload(). Every member must supply the same dtype and extent. Supported dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64. Scalar inputs are unsupported.

  • mode (str, optional) – Compile-time conversion, default "striped_to_blocked". All groups also support "blocked_to_striped". Blocks additionally support "blocked_to_warp_striped" and "warp_striped_to_blocked", which stripe within each physical warp and require a block size divisible by 32. Block scatter modes are "scatter_to_blocked", "scatter_to_striped", "scatter_to_striped_guarded", and "scatter_to_striped_flagged". Physical and logical warps support only "striped_to_blocked" and "blocked_to_striped"; their enclosing block must contain complete physical warps.

  • ranks (ThreadData, CuTe register tensor, or TensorSSA, optional) – Destination ranks, required for every scatter mode and rejected for layout conversions. Must have a signed integer dtype and the same per-thread extent as value. Valid ranks must be distinct across the block and lie in [0, block_tile_size). Guarded scatter skips negative ranks. Other scatter modes require each participating item’s rank to be in range. None is the default.

  • valid_flags (ThreadData, CuTe register tensor, or TensorSSA, optional) – Per-item flags, required for "scatter_to_striped_flagged" and rejected for other modes. Must have an integer, non-boolean dtype and the same extent as value. Only items with nonzero flags scatter; their ranks must be distinct and in range. None is the default.

  • warp_time_slicing (bool, optional) – Compile-time option, default False. Reuse block exchange scratch across warps to reduce shared memory usage at the cost of additional synchronization. Requires a block; guarded and flagged scatter do not support it.

Returns:

New writable payload with the input dtype, extent, and requested alignment in the output layout. All inputs, ranks, and flags remain unchanged. A scatter destination with no valid input is undefined; initialize it before reading it.

Return type:

cuda.coop.cutlass.ThreadData

Notes

The operation does not check rank uniqueness or bounds. Guarded scatter skips only negative ranks, and flagged scatter skips only items with a zero flag. Duplicate ranks on written items give undefined results. An out-of-range rank on a written item can write past the exchange scratch.

Scratch allocation and trailing synchronization are automatic. See Scratch allocation and reuse for storage reuse and Groups and tiles for participation requirements.

See also

cub::BlockExchange, cub::WarpExchange

C++ exchange primitives.

Examples

Reverse a 128-item block tile by assigning each input its destination rank. Each of the 64 threads holds two items.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as cutlass_coop

@cute.kernel
def reverse_tile(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = cutlass_coop.this_block()
    thread = block.rank()
    items = cutlass_coop.load(
        block, source, items_per_thread=items_per_thread
    )
    ranks = cutlass_coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        ranks[item] = cutlass.Int32(
            64 * items_per_thread - 1 - (thread * items_per_thread + item)
        )
    reversed_items = cutlass_coop.exchange(
        block, items, mode="scatter_to_blocked", ranks=ranks
    )
    cutlass_coop.store(block, destination, reversed_items)

@cute.jit
def launch(
    source: cute.Pointer,
    destination: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    reverse_tile(source, destination, items_per_thread).launch(
        grid=1, block=64
    )

cuda.coop.cutlass.shuffle(
group: ThreadGroup,
value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
/,
*,
mode: Literal['down', 'up', 'offset', 'rotate'] = 'down',
distance: Literal[1] | int | integer | CompilerIntegerLike = 1,
) → ThreadData | _ItemT#

Shift register payloads or select another block member’s scalar.

Payload shifts follow flattened blocked order: each thread’s items are consecutive, and a shift crosses thread boundaries as needed. Scalar offset and rotate calls select one other member’s value.

Parameters:
  • group (ThreadGroup) – Complete block whose members all call the primitive. Warp, mapped groups, clusters, and grids are unsupported.

  • value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Fixed-size payloads may be ThreadData, CuTe register tensors, or TensorSSA values. The latter two are converted with cuda.coop.cutlass.ThreadData.from_payload(). For scalar offset or rotate modes, each thread supplies one numeric scalar. All members use the same dtype and extent. Supported dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64.

  • mode (str, optional) – Compile-time mode, default "down". Payloads support only "up" and "down"; the first or last item of the block tile, respectively, is undefined. Up moves input i - 1 to output i; down moves input i + 1 to output i. Scalars require "offset" or "rotate": rank r receives the value from rank r + distance. Offset results are undefined when that source is outside the block; rotate wraps the source rank modulo the block size.

  • distance (int or CuTe integer scalar, optional) – Default 1. Payload shifts require a compile-time unit distance. Scalar modes accept compile-time or runtime distances, which may differ between threads. Offset distances must fit a signed 32-bit integer and may be zero or negative. Rotate requires at least two threads and 1 <= distance < block_size. A runtime distance may have a signed integer dtype up to 64 bits or an unsigned integer dtype up to 32 bits, but its value must remain in the mode’s allowed range.

Returns:

Scalar input produces a CuTe scalar with the input dtype. Payload input produces a new writable ThreadData with the input dtype, extent, and requested alignment. The input remains unchanged. Observe the chosen mode’s boundary rules before reading the result.

Return type:

CuTe numeric scalar or cuda.coop.cutlass.ThreadData

Notes

Scratch allocation and trailing synchronization are automatic. Every block member must call, including threads with out-of-range offset sources. Warp and logical-warp groups are not supported.

See also

cub::BlockShuffle

C++ shift, offset, and rotate primitive.

Examples

Rotate one scalar per thread by seven positions in a 64-thread block, including wraparound.

from cutlass import cute

import cuda.coop.cutlass as cutlass_coop

@cute.kernel
def rotate_tile(source: cute.Pointer, destination: cute.Pointer):
    block = cutlass_coop.this_block()
    thread = block.rank()
    inputs = cute.make_tensor(source, cute.make_layout(64))
    outputs = cute.make_tensor(destination, cute.make_layout(64))
    outputs[thread] = cutlass_coop.shuffle(
        block, inputs[thread], mode="rotate", distance=7
    )

@cute.jit
def launch(source: cute.Pointer, destination: cute.Pointer):
    rotate_tile(source, destination).launch(grid=1, block=64)

Comparison sorting#

cuda.coop.cutlass.merge_sort_keys(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
descending: bool = False,
valid_items: int | integer | CompilerIntegerLike | None = None,
oob_default: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Sort keys across a block or warp without changing the input.

Accepts readable ThreadData and CuTe register payloads. Each thread contributes the same fixed number of items in blocked order, and all members of the group must participate.

Parameters:
  • group (ThreadGroup) – this_block(), this_warp(), or this_warp().group_by(width). Blocks require a power-of-two total thread count and may be multidimensional. Logical warp widths are 1, 2, 4, 8, 16, and 32; their enclosing physical warps must be complete.

  • keys (ThreadData or CuTe register payload) – Per-thread keys in blocked order, with a positive, compile-time extent. Read-only payloads are accepted. CuTe register-memory tensors and TensorSSA values are converted through ThreadData.from_payload. Supported element types are signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64.

  • descending (bool, optional) – Compile-time order selector. The default is ascending order. Custom comparison predicates are not supported.

  • valid_items (integer, optional) – Group-uniform count in [0, group_size * items_per_thread]. Only this blocked input prefix participates in the sort. Supply oob_default together with this argument for a partial tile. Runtime counts accept signed integer types up to 64 bits or unsigned integer types up to 32 bits. Invalid static counts fail compilation; invalid runtime counts trap before conversion to CUB’s count type.

  • oob_default (scalar, optional) – Group-uniform sentinel that sorts after valid keys: an upper bound for ascending order or a lower bound for descending order. A runtime CuTe scalar must match the key type exactly. Representable Python numeric literals are converted to the key type.

  • temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Warp groups always use automatic storage with independent slices per group. If the descriptor has auto_sync=False, synchronize the block before reusing its storage.

Returns:

Fresh sorted keys with the input element type and per-thread extent, including when the input is a CuTe register payload. Output is blocked: thread t owns sorted positions t * items_per_thread + i. Only the first valid_items positions are defined for a partial tile. Equal keys have no stability guarantee.

Return type:

ThreadData

Examples

Sort a partial tile in ascending order and its key/index pairs in descending order. Stores write only the valid output prefix.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def sort_prefix(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    valid_items = 64 * items_per_thread - 3
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    indices = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        indices[item] = cutlass.Int32(
            block.rank() * items_per_thread + item
        )
    scratch = coop.TempStorage(auto_sync=True)
    ordered = coop.merge_sort_keys(
        block,
        keys,
        valid_items=valid_items,
        oob_default=1000,
        temp_storage=scratch,
    )
    pair_keys, pair_indices = coop.merge_sort_pairs(
        block,
        keys,
        indices,
        descending=True,
        valid_items=valid_items,
        oob_default=-1000,
        temp_storage=scratch,
    )
    coop.store(block, ascending, ordered, valid_items=valid_items)
    coop.store(block, descending, pair_keys, valid_items=valid_items)
    coop.store(block, positions, pair_indices, valid_items=valid_items)

@cute.jit
def launch(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    sort_prefix(
        source, ascending, descending, positions, items_per_thread
    ).launch(grid=1, block=64)

cuda.coop.cutlass.merge_sort_pairs(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
descending: bool = False,
valid_items: int | integer | CompilerIntegerLike | None = None,
oob_default: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
temp_storage: TempStorageLike | None = None,
) → tuple[ThreadData, ThreadData]#

Sort keys and their associated values without changing either input.

Accepts readable ThreadData and CuTe register payloads for either operand. Group participation and ordering follow cuda.coop.cutlass.merge_sort_keys().

Parameters:
  • group (ThreadGroup) – Complete block, physical warp, or supported logical warp. Block thread counts must be powers of two; enclosing physical warps must be complete for warp operations.

  • keys (ThreadData or CuTe register payload) – Fixed-size per-thread payloads with equal extents, in blocked order. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64 are supported.

  • values (ThreadData or CuTe register payload) – Fixed-size per-thread payloads with equal extents, in blocked order. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64 are supported.

  • descending (bool, optional) – Compile-time order selector. The default is ascending key order. Custom comparison predicates are not supported.

  • valid_items (integer, optional) – Group-uniform length of the valid blocked input prefix, from zero through the group tile size. Supply it together with oob_default. Count types and runtime checks match merge_sort_keys.

  • oob_default (scalar, optional) – Key sentinel that sorts after valid keys. Runtime CuTe scalars must have the key type; representable Python literals are converted to it.

  • temp_storage (TempStorage, optional) – Explicit scratch for block groups only. Warp storage is automatic. Retain automatic synchronization or synchronize before scratch reuse.

Returns:

Fresh sorted keys and corresponding values, each retaining its own input element type and extent. CuTe register inputs also return ThreadData. Only the valid blocked prefix is defined for a partial tile. Key/value associations are preserved; equal keys have no stability guarantee.

Return type:

tuple[ThreadData, ThreadData]

Examples

Carry original indices through a descending partial-tile sort. The ascending keys-only call uses the same automatically synchronized scratch descriptor.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def sort_prefix(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    valid_items = 64 * items_per_thread - 3
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    indices = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        indices[item] = cutlass.Int32(
            block.rank() * items_per_thread + item
        )
    scratch = coop.TempStorage(auto_sync=True)
    ordered = coop.merge_sort_keys(
        block,
        keys,
        valid_items=valid_items,
        oob_default=1000,
        temp_storage=scratch,
    )
    pair_keys, pair_indices = coop.merge_sort_pairs(
        block,
        keys,
        indices,
        descending=True,
        valid_items=valid_items,
        oob_default=-1000,
        temp_storage=scratch,
    )
    coop.store(block, ascending, ordered, valid_items=valid_items)
    coop.store(block, descending, pair_keys, valid_items=valid_items)
    coop.store(block, positions, pair_indices, valid_items=valid_items)

@cute.jit
def launch(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    sort_prefix(
        source, ascending, descending, positions, items_per_thread
    ).launch(grid=1, block=64)

Radix sorting and ranking#

cuda.coop.cutlass.radix_sort_keys(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample | _KeyT,
/,
*,
begin_bit: int | integer | CompilerIntegerLike = 0,
end_bit: int | integer | CompilerIntegerLike | None = None,
descending: bool = False,
temp_storage: TempStorageLike | None = None,
blocked_to_striped: bool = False,
) → ThreadData | _KeyT#

Return stable radix-sorted keys without changing the input.

Accepts scalars and readable or CuTe register payloads, including floating-point keys. Output may use blocked or striped ownership. Every block thread participates with the same controls and per-thread extent.

Parameters:
  • group (ThreadGroup) – Complete physical block from this_block(). Multidimensional blocks are supported. The block tile must contain at most 65,535 items.

  • keys (scalar, ThreadData, or CuTe register payload) – Keys in blocked order, with element type Int32, Uint32, Int64, Uint64, Float32, or Float64. A scalar contributes one key per thread. Register-memory tensors and TensorSSA values are converted through ThreadData.from_payload; read-only payloads are also accepted.

  • begin_bit (integer, optional) – Block-uniform half-open interval in CUB’s transformed key bits, with 0 <= begin_bit < end_bit <= key_width. Begin defaults to zero; omitted end selects the key width. Either bound may be a runtime CuTe integer, signed up to 64 bits or unsigned up to 32 bits. Invalid runtime bounds trap before narrowing to CUB’s integer arguments.

  • end_bit (integer, optional) – Block-uniform half-open interval in CUB’s transformed key bits, with 0 <= begin_bit < end_bit <= key_width. Begin defaults to zero; omitted end selects the key width. Either bound may be a runtime CuTe integer, signed up to 64 bits or unsigned up to 32 bits. Invalid runtime bounds trap before narrowing to CUB’s integer arguments.

  • descending (bool, optional) – Compile-time selector for descending order. The default is ascending.

  • temp_storage (TempStorage, optional) – Explicit block scratch, otherwise allocated automatically. Requested alignment is a minimum. With auto_sync=False, synchronize the block before reusing the storage.

  • blocked_to_striped (bool, optional) – Compile-time output-layout selector. The default is blocked output, where thread t owns sorted indices t * items_per_thread + i. True returns striped output, with indices i * block_threads + t. Use a matching striped Store to write those registers in sorted order.

Returns:

Sorted keys with the input element type. Scalar inputs return a CuTe scalar; payload inputs return fresh ThreadData of the same extent, including when the input uses a CuTe register representation. Equal selected digits retain blocked input order in either direction.

Return type:

CuTe scalar or ThreadData

Notes

CUB transforms keys before selecting the requested bits. Signed integers invert their sign bit. Floating-point keys invert all bits when negative and only the sign bit otherwise. Returned keys keep their original bits. Positive and negative zero compare equally; NaNs follow the transformed bit order, with no guarantee of numeric ordering.

Examples

Sort all key bits in ascending order, then sort key/index pairs by their low four bits in descending order. The latter produces striped registers, so its stores use algorithm="striped".

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def sort_digits(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    indices = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        indices[item] = cutlass.Int32(
            block.rank() * items_per_thread + item
        )
    ordered = coop.radix_sort_keys(block, keys)
    pair_keys, pair_indices = coop.radix_sort_pairs(
        block,
        keys,
        indices,
        begin_bit=0,
        end_bit=4,
        descending=True,
        blocked_to_striped=True,
    )
    coop.store(block, ascending, ordered)
    coop.store(block, descending, pair_keys, algorithm="striped")
    coop.store(block, positions, pair_indices, algorithm="striped")

@cute.jit
def launch(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    sort_digits(
        source, ascending, descending, positions, items_per_thread
    ).launch(grid=1, block=64)

cuda.coop.cutlass.radix_sort_pairs(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample | _KeyT,
values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample | _ValueT,
/,
*,
begin_bit: int | integer | CompilerIntegerLike = 0,
end_bit: int | integer | CompilerIntegerLike | None = None,
descending: bool = False,
temp_storage: TempStorageLike | None = None,
blocked_to_striped: bool = False,
) → tuple[ThreadData, ThreadData] | tuple[_KeyT, _ValueT]#

Return stable radix-sorted keys and their associated values.

Accepts scalars and readable or CuTe register payloads, including floating-point keys. Output may use blocked or striped ownership. Neither input is modified. Key transformations and stable tie handling follow cuda.coop.cutlass.radix_sort_keys().

Parameters:
  • group (ThreadGroup) – Complete physical block. Every thread participates with the same controls; the block tile contains at most 65,535 items.

  • keys (scalar, ThreadData, or CuTe register payload) – Matching scalar or fixed-size payload shapes. Payloads have equal per-thread extents and use blocked input order. Key types are Int32, Uint32, Int64, Uint64, Float32, and Float64. Values may independently use signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, or Float64. Read-only and CuTe register payloads are accepted; one operand may be ThreadData while the other is a register payload of the same extent.

  • values (scalar, ThreadData, or CuTe register payload) – Matching scalar or fixed-size payload shapes. Payloads have equal per-thread extents and use blocked input order. Key types are Int32, Uint32, Int64, Uint64, Float32, and Float64. Values may independently use signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, or Float64. Read-only and CuTe register payloads are accepted; one operand may be ThreadData while the other is a register payload of the same extent.

  • begin_bit (integer, optional) – Block-uniform half-open interval in transformed key bits. Begin defaults to zero and end to the key width. Runtime integer bounds and validation follow radix_sort_keys.

  • end_bit (integer, optional) – Block-uniform half-open interval in transformed key bits. Begin defaults to zero and end to the key width. Runtime integer bounds and validation follow radix_sort_keys.

  • descending (bool, optional) – Compile-time order selector. The default is ascending key order.

  • temp_storage (TempStorage, optional) – Explicit block scratch. Omit for automatic allocation. Synchronize before reuse if its descriptor specifies auto_sync=False.

  • blocked_to_striped (bool, optional) – Compile-time selector for striped output, applied to both keys and values. The default is blocked output. Store both results with the corresponding layout.

Returns:

Sorted keys and associated values, each retaining its input element type. Scalar operands return CuTe scalars; payload operands return fresh ThreadData with the original per-thread extent. Associations are preserved, and equal selected digits retain blocked input order.

Return type:

tuple

Examples

Stably order key/index pairs by their low four bits in descending order. Match striped output registers with striped stores.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def sort_digits(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    indices = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        indices[item] = cutlass.Int32(
            block.rank() * items_per_thread + item
        )
    ordered = coop.radix_sort_keys(block, keys)
    pair_keys, pair_indices = coop.radix_sort_pairs(
        block,
        keys,
        indices,
        begin_bit=0,
        end_bit=4,
        descending=True,
        blocked_to_striped=True,
    )
    coop.store(block, ascending, ordered)
    coop.store(block, descending, pair_keys, algorithm="striped")
    coop.store(block, positions, pair_indices, algorithm="striped")

@cute.jit
def launch(
    source: cute.Pointer,
    ascending: cute.Pointer,
    descending: cute.Pointer,
    positions: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    sort_digits(
        source, ascending, descending, positions, items_per_thread
    ).launch(grid=1, block=64)

cuda.coop.cutlass.radix_rank_keys(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_RankKeyT] | CutlassTensorSample | CutlassTensorSSASample | _RankKeyT,
/,
*,
begin_bit: int = 0,
end_bit: int | None = None,
radix_bits: int | None = None,
descending: bool = False,
exclusive_digit_prefix: ThreadDataLike[int32] | ThreadDataLike[CompilerIntegerLike] | None = None,
) → ThreadData | CompilerIntegerLike#

Return stable digit ranks and optional exclusive bin prefixes.

Accepts scalar keys and readable or CuTe register payloads. An optional writable payload receives bin prefixes. Keys remain unchanged.

Parameters:
  • group (ThreadGroup) – Complete physical block. Every thread participates with identical compile-time controls and payload extents. Multidimensional blocks are supported; the block tile contains at most 65,535 items.

  • keys (scalar, ThreadData, or CuTe register payload) – Int32, Uint32, Int64, or Uint64 keys in blocked order. A scalar contributes one key per thread. CuTe register-memory tensors, TensorSSA values, and read-only payloads are accepted. Signed keys invert their sign bit before digit extraction.

  • begin_bit (int, optional) – Compile-time half-open interval within the key width, selecting one through eight bits. Begin defaults to zero. Omitted end is begin plus radix_bits, or begin plus four if both are omitted.

  • end_bit (int, optional) – Compile-time half-open interval within the key width, selecting one through eight bits. Begin defaults to zero. Omitted end is begin plus radix_bits, or begin plus four if both are omitted.

  • radix_bits (int, optional) – Compile-time digit width. If end_bit is explicit, this must equal end_bit - begin_bit.

  • descending (bool, optional) – Compile-time selector for descending digit order. The default is ascending order.

  • exclusive_digit_prefix (ThreadData, optional) – Writable Int32 side output, distinct from keys. An omitted dtype is inferred as Int32. Each thread provides P = max(1, ceil(2**radix_bits / block_threads)) items. Thread t owns ascending bin indices t * P + i in either ordering direction. A bin’s prefix counts keys with smaller digits for ascending order or greater digits for descending order. Slots past the last bin are undefined and must not be read or stored. Register tensors cannot be passed directly as this output.

Returns:

Zero-based ranks with fixed Int32 element type. Scalar input returns a CuTe scalar; payload input returns fresh ThreadData with the input extent. Equal digits retain blocked input order. Element i at thread t receives the sorted position of the original key at blocked index t * items_per_thread + i.

Return type:

Int32 or ThreadData

Notes

Scratch allocation and synchronization before reuse are automatic.

Examples

Rank the low four bits and retrieve the starting offset of each of the sixteen digit bins. Only the sixteen defined prefix slots are stored.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def rank_digits(
    source: cute.Pointer,
    ranks_output: cute.Pointer,
    prefixes_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    prefixes = coop.ThreadData(items_per_thread=1)
    ranks = coop.radix_rank_keys(
        block, keys, radix_bits=4, exclusive_digit_prefix=prefixes
    )
    coop.store(block, ranks_output, ranks)
    coop.store(block, prefixes_output, prefixes, valid_items=16)

@cute.jit
def launch(
    source: cute.Pointer,
    ranks_output: cute.Pointer,
    prefixes_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    rank_digits(
        source, ranks_output, prefixes_output, items_per_thread
    ).launch(grid=1, block=64)

Top-k selection#

cuda.coop.cutlass.topk_min_keys(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Select the block’s smallest keys without changing the input.

Accepts readable ThreadData and CuTe register payloads. All block threads participate, including those whose input items are outside the valid prefix.

Parameters:
  • group (ThreadGroup) – Complete one-dimensional block from this_block().

  • keys (ThreadData or CuTe register payload) – Per-thread keys in blocked order with a positive, compile-time extent. Read-only payloads, CuTe register-memory tensors, and TensorSSA values are accepted. Supported element types are signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64. Register inputs are converted through ThreadData.from_payload.

  • k (integer) – Block-uniform number of requested items in [0, N], where N = block_threads * items_per_thread. May be static or runtime.

  • valid_items (integer, optional) – Block-uniform length of the valid blocked input prefix, also in [0, N]. Omit it to use the entire tile. If k > valid_items, all valid items are selected.

  • temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Requested alignment is a minimum. With auto_sync=False, synchronize the block before reusing the storage.

Returns:

Fresh selected keys with the input element type and per-thread extent, including for CuTe register inputs. Only the first min(k, valid_items) blocked output positions are defined; omit valid_items to use N in this expression. Thread t owns positions t * items_per_thread + i. Remaining positions must not be read or stored. The selected keys are unsorted.

Return type:

ThreadData

Notes

Selection and ordering among equal keys are unspecified. Positive and negative floating-point zero compare equally, and selected keys retain their original bits. NaNs have no guaranteed numeric ordering.

Runtime k and valid_items accept signed integers up to 64 bits or unsigned integers up to 32 bits. Invalid static counts fail compilation; invalid runtime counts trap before conversion to CUB’s count type. Zero k or valid_items leaves no defined output positions.

Examples

Select the seven smallest and seven largest keys. All threads participate, and each store writes only the seven defined positions. Neither selection sorts its output.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def select_extremes(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    low = coop.topk_min_keys(block, keys, k=7)
    high = coop.topk_max_keys(block, keys, k=7)
    # Selection is unsorted; only the first seven positions are defined.
    coop.store(block, smallest, low, valid_items=7)
    coop.store(block, largest, high, valid_items=7)

@cute.jit
def launch(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    select_extremes(source, smallest, largest, items_per_thread).launch(
        grid=1, block=64
    )

cuda.coop.cutlass.topk_max_keys(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Select the block’s largest keys without changing the input.

Accepts the same inputs and controls as cuda.coop.cutlass.topk_min_keys(), selecting the largest keys. All threads in the block must participate.

Parameters:
  • group (ThreadGroup) – Complete one-dimensional block from this_block().

  • keys (ThreadData or CuTe register payload) – Fixed-size per-thread keys in blocked order. Read-only payloads, register-memory tensors, and TensorSSA values are accepted. The element types are those supported by topk_min_keys.

  • k (integer) – Block-uniform count in [0, block_threads * items_per_thread]. Static and runtime counts are accepted.

  • valid_items (integer, optional) – Block-uniform length of the valid input prefix. Defaults to the whole tile and has the same range as k.

  • temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Synchronize before reuse if its descriptor specifies auto_sync=False.

Returns:

Fresh selected keys with the input element type and extent. Only the first min(k, valid_items) blocked positions are defined, or min(k, N) when valid_items is omitted, where N = block_threads * items_per_thread. The selected keys are unsorted; remaining positions must not be consumed. Ties, floating-point ordering, and count validation follow topk_min_keys.

Return type:

ThreadData

Examples

Select the seven largest and seven smallest keys, storing only defined output positions. The selected keys are unsorted.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def select_extremes(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    low = coop.topk_min_keys(block, keys, k=7)
    high = coop.topk_max_keys(block, keys, k=7)
    # Selection is unsorted; only the first seven positions are defined.
    coop.store(block, smallest, low, valid_items=7)
    coop.store(block, largest, high, valid_items=7)

@cute.jit
def launch(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    select_extremes(source, smallest, largest, items_per_thread).launch(
        grid=1, block=64
    )

cuda.coop.cutlass.topk_min_pairs(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
temp_storage: TempStorageLike | None = None,
) → tuple[ThreadData, ThreadData]#

Select the smallest keys and their associated values.

Accepts readable ThreadData and CuTe register payloads for either operand. Both inputs remain unchanged. Participation, count validation, and key ordering follow cuda.coop.cutlass.topk_min_keys().

Parameters:
  • group (ThreadGroup) – Complete one-dimensional block. All block threads must participate.

  • keys (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64 are supported.

  • values (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers, Float32, and Float64 are supported.

  • k (integer) – Block-uniform count in [0, block_threads * items_per_thread]. Static and runtime counts are accepted.

  • valid_items (integer, optional) – Block-uniform length of the valid blocked input prefix. Defaults to the entire tile and has the same range as k. If k exceeds this count, all valid key/value pairs are selected.

  • temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Synchronize before reuse if its descriptor specifies auto_sync=False.

Returns:

Fresh selected keys and associated values, each retaining its input element type and extent. Only the first min(k, valid_items) blocked positions are defined, or min(k, N) when valid_items is omitted, where N = block_threads * items_per_thread. Each selected value stays with its original key. Results are unsorted, with no stability guarantee for ties; undefined tail positions must not be read or stored.

Return type:

tuple[ThreadData, ThreadData]

Examples

Select seven minimum and maximum key/index pairs from a partial input tile. Original indices remain associated with their selected keys.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def select_pairs(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    smallest_indices: cute.Pointer,
    largest_indices: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    valid_items = 64 * items_per_thread - 3
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    indices = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        indices[item] = cutlass.Int32(
            block.rank() * items_per_thread + item
        )
    low, low_indices = coop.topk_min_pairs(
        block, keys, indices, k=7, valid_items=valid_items
    )
    high, high_indices = coop.topk_max_pairs(
        block, keys, indices, k=7, valid_items=valid_items
    )
    coop.store(block, smallest, low, valid_items=7)
    coop.store(block, largest, high, valid_items=7)
    coop.store(block, smallest_indices, low_indices, valid_items=7)
    coop.store(block, largest_indices, high_indices, valid_items=7)

@cute.jit
def launch(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    smallest_indices: cute.Pointer,
    largest_indices: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    select_pairs(
        source,
        smallest,
        largest,
        smallest_indices,
        largest_indices,
        items_per_thread,
    ).launch(grid=1, block=64)

cuda.coop.cutlass.topk_max_pairs(
group: ThreadGroup,
keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
temp_storage: TempStorageLike | None = None,
) → tuple[ThreadData, ThreadData]#

Select the largest keys and their associated values.

Accepts the same inputs and controls as cuda.coop.cutlass.topk_min_pairs(), selecting the largest keys. Both inputs remain unchanged.

Parameters:
  • group (ThreadGroup) – Complete one-dimensional block. All block threads must participate.

  • keys (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Read-only payloads and CuTe register representations are accepted. Each operand independently uses a numeric element type supported by topk_min_pairs.

  • values (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Read-only payloads and CuTe register representations are accepted. Each operand independently uses a numeric element type supported by topk_min_pairs.

  • k (integer) – Block-uniform count in [0, block_threads * items_per_thread]. Static and runtime counts are accepted.

  • valid_items (integer, optional) – Block-uniform length of the valid input prefix. Defaults to the entire tile and has the same range as k. If k exceeds this count, all valid key/value pairs are selected.

  • temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Synchronize before reuse if its descriptor specifies auto_sync=False.

Returns:

Fresh selected keys and associated values, retaining each input’s element type and extent. Only the first min(k, valid_items) blocked positions are defined, or min(k, N) when valid_items is omitted, where N = block_threads * items_per_thread. Associations are preserved. Results are unsorted, with no stability guarantee for ties. Floating-point ordering, count validation, and undefined tails follow cuda.coop.cutlass.topk_min_keys().

Return type:

tuple[ThreadData, ThreadData]

Examples

Select seven maximum and minimum key/index pairs from a partial input tile. Stores exclude the undefined output tail.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def select_pairs(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    smallest_indices: cute.Pointer,
    largest_indices: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    valid_items = 64 * items_per_thread - 3
    keys = coop.load(block, source, items_per_thread=items_per_thread)
    indices = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        indices[item] = cutlass.Int32(
            block.rank() * items_per_thread + item
        )
    low, low_indices = coop.topk_min_pairs(
        block, keys, indices, k=7, valid_items=valid_items
    )
    high, high_indices = coop.topk_max_pairs(
        block, keys, indices, k=7, valid_items=valid_items
    )
    coop.store(block, smallest, low, valid_items=7)
    coop.store(block, largest, high, valid_items=7)
    coop.store(block, smallest_indices, low_indices, valid_items=7)
    coop.store(block, largest_indices, high_indices, valid_items=7)

@cute.jit
def launch(
    source: cute.Pointer,
    smallest: cute.Pointer,
    largest: cute.Pointer,
    smallest_indices: cute.Pointer,
    largest_indices: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    select_pairs(
        source,
        smallest,
        largest,
        smallest_indices,
        largest_indices,
        items_per_thread,
    ).launch(grid=1, block=64)

Neighbor comparisons#

cuda.coop.cutlass.adjacent_difference(
group: ThreadGroup,
values: _ReadableThreadDataLike[_T] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
direction: Literal['left', 'right'] = 'left',
valid_items: int | integer | CompilerIntegerLike | None = None,
tile_predecessor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
tile_successor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Return blocked neighbor differences, preserving the input payload.

Parameters:
  • group (ThreadGroup) – A complete this_block() group. Every member participates in the same call; multidimensional blocks use linear thread-rank order.

  • values (ThreadData or CuTe register payload) – Readable fixed-size payload in blocked order. Signed and unsigned 8-, 16-, 32-, and 64-bit integers, float32, and float64 are supported. Initialize every input slot, including any invalid suffix. CuTe register tensors and TensorSSA values are adapted through ThreadData.from_payload.

  • direction ({"left", "right"}, optional) – Compile-time choice, default "left". Subtract the previous item from the current item for left differences, or the next item from the current item for right differences.

  • valid_items (integer, optional) – Block-uniform count in [0, block_size * items_per_thread]. Omit it to process the full tile. The suffix beyond this count is copied unchanged. For right differences, this argument cannot be combined with tile_successor_item.

  • tile_predecessor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Left differences accept only a predecessor; right differences accept only a successor. Without that neighbor, the boundary input is copied unchanged.

  • tile_successor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Left differences accept only a predecessor; right differences accept only a successor. Without that neighbor, the boundary input is copied unchanged.

  • temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Omit it for automatic storage. With auto_sync=False, synchronize the block before reusing the descriptor in another primitive.

Returns:

Fresh values with the input dtype, extent, and minimum alignment. Invalid suffix items and a boundary without an external neighbor retain their input values. Only built-in subtraction is supported.

Return type:

cuda.coop.cutlass.ThreadData

Notes

tile_predecessor_item and tile_successor_item given as NumPy or runtime scalars must match the input dtype. Plain Python integers must fit the input dtype; out-of-range values raise ValueError. Python floats require a floating-point input dtype. Finite values must fit its range; infinities and NaNs are accepted.

valid_items may be a Python integer, a runtime signed integer up to 64 bits, or a runtime unsigned integer up to 32 bits. An out-of-range static count raises ValueError when the kernel compiles. An out-of-range runtime count stops the kernel with a trap before conversion to CUB’s int count.

See the Adjacent Difference visualization for tile boundaries.

Examples

Compute left differences using zero as the tile predecessor, then mark run heads and tails. Scratch is synchronized automatically before reuse by the second collective.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def compare_neighbors(
    source: cute.Pointer,
    differences_output: cute.Pointer,
    heads_output: cute.Pointer,
    tails_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    scratch = coop.TempStorage(auto_sync=True)
    differences = coop.adjacent_difference(
        block, values, tile_predecessor_item=0, temp_storage=scratch
    )
    heads, tails = coop.discontinuity(
        block, values, mode="heads_and_tails", temp_storage=scratch
    )
    coop.store(block, differences_output, differences)
    coop.store(block, heads_output, heads)
    coop.store(block, tails_output, tails)

@cute.jit
def launch(
    source: cute.Pointer,
    differences_output: cute.Pointer,
    heads_output: cute.Pointer,
    tails_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    compare_neighbors(
        source,
        differences_output,
        heads_output,
        tails_output,
        items_per_thread,
    ).launch(grid=1, block=64)

cuda.coop.cutlass.discontinuity(
group: ThreadGroup,
values: _ReadableThreadDataLike[_T] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
mode: Literal['heads', 'tails', 'heads_and_tails'] = 'heads',
tile_predecessor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
tile_successor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
temp_storage: TempStorageLike | None = None,
) → ThreadData | tuple[ThreadData, ThreadData]#

Flag unequal neighbors in a full blocked tile.

Parameters:
  • group (ThreadGroup) – A complete this_block() group. Every member participates in the same call; multidimensional blocks use linear thread-rank order.

  • values (ThreadData or CuTe register payload) – Readable fixed-size payload in blocked order. Signed and unsigned 8-, 16-, 32-, and 64-bit integers, float32, and float64 are supported. CuTe register tensors and TensorSSA values are adapted through ThreadData.from_payload.

  • mode ({"heads", "tails", "heads_and_tails"}, optional) – Compile-time result selection, default "heads". Heads compare the previous item with the current item; tails compare the current item with the next item. Unequal neighbors produce one, otherwise zero.

  • tile_predecessor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Heads accept a predecessor, tails accept a successor, and "heads_and_tails" accepts both. Without a predecessor the first head is one; without a successor the last tail is one.

  • tile_successor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Heads accept a predecessor, tails accept a successor, and "heads_and_tails" accepts both. Without a predecessor the first head is one; without a successor the last tail is one.

  • temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Omit it for automatic storage. With auto_sync=False, synchronize the block before reusing the descriptor in another primitive.

Returns:

Fresh int32 heads or tails, or (heads, tails). Each result has the input extent and minimum alignment. The input remains unchanged.

Return type:

cuda.coop.cutlass.ThreadData or tuple of ThreadData

Notes

Partial tiles are unsupported. Padding participates in comparisons and can change the last valid tail. Only built-in inequality is supported.

tile_predecessor_item and tile_successor_item given as NumPy or runtime scalars must match the input dtype. Plain Python integers must fit the input dtype; out-of-range values raise ValueError. Python floats require a floating-point input dtype. Finite values must fit its range; infinities and NaNs are accepted.

See the Discontinuity visualization for head and tail flags.

Examples

Mark both ends of each equal-value run and compute adjacent differences over the same tile.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def compare_neighbors(
    source: cute.Pointer,
    differences_output: cute.Pointer,
    heads_output: cute.Pointer,
    tails_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.load(block, source, items_per_thread=items_per_thread)
    scratch = coop.TempStorage(auto_sync=True)
    differences = coop.adjacent_difference(
        block, values, tile_predecessor_item=0, temp_storage=scratch
    )
    heads, tails = coop.discontinuity(
        block, values, mode="heads_and_tails", temp_storage=scratch
    )
    coop.store(block, differences_output, differences)
    coop.store(block, heads_output, heads)
    coop.store(block, tails_output, tails)

@cute.jit
def launch(
    source: cute.Pointer,
    differences_output: cute.Pointer,
    heads_output: cute.Pointer,
    tails_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    compare_neighbors(
        source,
        differences_output,
        heads_output,
        tails_output,
        items_per_thread,
    ).launch(grid=1, block=64)

Histogram#

cuda.coop.cutlass.histogram(
group: ThreadGroup,
samples: _ReadableThreadDataLike[int | uint8 | int32 | uint32 | int64 | uint64 | CompilerIntegerLike] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
bins: int,
bins_per_thread: int = 1,
counter_dtype: type[int | _Counter] | dtype | None = None,
algorithm: Literal['atomic', 'sort'] = 'atomic',
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Count integer samples into fresh striped bin counters.

Parameters:
  • group (ThreadGroup) – A complete one-dimensional block. Every thread participates.

  • samples (ThreadData or CuTe register payload) – Readable fixed-size bin indices in [0, bins). Supports uint8, int32, uint32, int64, and uint64. CuTe register tensors and immutable vectors are copied through ThreadData.from_payload.

  • bins (int) – Positive compile-time number of bins.

  • bins_per_thread (int, optional) – Positive compile-time result extent. block_size * bins_per_thread must cover every bin. Defaults to one.

  • counter_dtype (object, optional) – Independent counter dtype: int32 (default), uint32, int64, or uint64. Accepts NumPy and CuTe selectors. Python int means int32.

  • algorithm ({"atomic", "sort"}, optional) – Compile-time algorithm. Both preserve the input samples.

  • temp_storage (TempStorage, optional) – Optional scratch for intermediate counters and CUB storage. Both algorithms allocate counters in shared memory, including "atomic". Omit the descriptor for automatic allocation. Requested alignment is a minimum. With auto_sync=False, synchronize the block before reusing the descriptor.

Returns:

Fresh counters in striped order: member t receives bin t + i * block_size in slot i. Slots beyond bins are zero. Use striped Store to write contiguous bin order.

Return type:

ThreadData

Notes

Each call starts from zero, even when scratch is reused. Accumulate returned counters explicitly to count several tiles. There is no valid_items control; padding contributes samples. Sample indices must lie in [0, bins); the operation does not check their bounds. See the Histogram visualization for sample and counter layouts.

Examples

Count samples with atomic and sort algorithms. Both return seventy counters followed by zero padding in striped register order. Each output allocation holds 128 counters for this 64-thread block.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def count_samples(
    source: cute.Pointer,
    atomic_output: cute.Pointer,
    sort_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    samples = coop.load(block, source, items_per_thread=items_per_thread)
    atomic_counts = coop.histogram(
        block, samples, bins=70, bins_per_thread=2
    )
    sort_counts = coop.histogram(
        block, samples, bins=70, bins_per_thread=2, algorithm="sort"
    )
    coop.store(block, atomic_output, atomic_counts, algorithm="striped")
    coop.store(block, sort_output, sort_counts, algorithm="striped")

@cute.jit
def launch(
    source: cute.Pointer,
    atomic_output: cute.Pointer,
    sort_output: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    count_samples(
        source, atomic_output, sort_output, items_per_thread
    ).launch(grid=1, block=64)

Run Length Decode#

cuda.coop.cutlass.run_length_decode(
group: ThreadGroup,
run_values: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
run_lengths: _ReadableThreadDataLike[_LengthT] | CutlassTensorSample | CutlassTensorSSASample,
/,
*,
decoded_items_per_thread: int,
decoded_window_offset: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] = 0,
temp_storage: TempStorageLike | None = None,
) → ThreadData#

Decode a blocked window without changing either input payload.

CuTe register-memory tensors and TensorSSA values are adapted through cuda.coop.cutlass.ThreadData.from_payload(). The window may start inside a run and may extend past the decoded stream.

Parameters:
  • group (ThreadGroup) – Complete one-dimensional block. Every thread must participate.

  • run_values (ThreadData or CuTe register payload) – Equal fixed-size per-thread payloads in blocked order. Values use the signed or unsigned 8-, 16-, 32-, or 64-bit integers, Float32, or Float64. Lengths use signed or unsigned integers up to 64 bits. Positive lengths precede all trailing zero padding; an all-zero tile represents an empty stream.

  • run_lengths (ThreadData or CuTe register payload) – Equal fixed-size per-thread payloads in blocked order. Values use the signed or unsigned 8-, 16-, 32-, or 64-bit integers, Float32, or Float64. Lengths use signed or unsigned integers up to 64 bits. Positive lengths precede all trailing zero padding; an all-zero tile represents an empty stream.

  • decoded_items_per_thread (int) – Positive compile-time output extent, independent of the run extent.

  • decoded_window_offset (integer, optional) – Block-uniform nonnegative start in the decoded stream, default zero. Member t starts at this offset plus t * decoded_items_per_thread. Static and runtime integers up to uint64 retain their width through validation. A window starting beyond the stream contains zeros.

  • temp_storage (TempStorage, optional) – Explicit block scratch, or omit for automatic allocation. Requested alignment is a minimum. Synchronize before reuse with auto_sync=False.

Returns:

Fresh payload with the run-value dtype and requested output extent. Slots beyond the decoded stream contain zero. Neither input is modified. The decoded total must fit uint32; negative lengths, misplaced zero padding, and overflow trap before decoding.

Return type:

ThreadData

Notes

Both run and window tile extents must fit signed 32-bit integers. Invalid static controls fail compilation; negative runtime offsets trap before writing outputs. Each call prepares its own CUB run table. Use cuda.coop.cutlass.run_length_decode_into() to write a full stream while preparing the table once.

See the Run Length Decode visualization for windows and zero-filled tails.

Examples

Decode a 64-item window starting inside a run, then write the full decoded stream at destination offset five. A window extending beyond the stream ends with zeros.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def decode_runs(
    window_pointer: cute.Pointer,
    stream_pointer: cute.Pointer,
    totals_pointer: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.ThreadData(items_per_thread)
    lengths = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        values[item] = cutlass.Int32(
            block.rank() * items_per_thread + item + 10
        )
        lengths[item] = cutlass.Uint32(2)
    scratch = coop.TempStorage(auto_sync=True)
    window = coop.run_length_decode(
        block,
        values,
        lengths,
        decoded_items_per_thread=2,
        decoded_window_offset=3,
        temp_storage=scratch,
    )
    coop.store(block, window_pointer, window)
    destination = cute.make_tensor(
        stream_pointer, cute.make_layout(64 * items_per_thread + 5)
    )
    total = coop.run_length_decode_into(
        block,
        values,
        lengths,
        destination,
        decoded_items_per_thread=1,
        destination_offset=5,
        temp_storage=scratch,
    )
    totals = cute.make_tensor(totals_pointer, cute.make_layout(32))
    totals[block.rank()] = total

@cute.jit
def launch(
    window_pointer: cute.Pointer,
    stream_pointer: cute.Pointer,
    totals_pointer: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    decode_runs(
        window_pointer, stream_pointer, totals_pointer, items_per_thread
    ).launch(grid=1, block=32)

cuda.coop.cutlass.run_length_decode_into(
group: ThreadGroup,
run_values: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
run_lengths: _ReadableThreadDataLike[_LengthT] | CutlassTensorSample | CutlassTensorSSASample,
destination: CutlassTensorSample,
/,
*,
decoded_items_per_thread: int,
destination_offset: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] = 0,
temp_storage: TempStorageLike | None = None,
) → CompilerIntegerLike#

Decode a full stream into a CuTe global-memory tensor.

The prepared CUB table remains live through all internal decode windows. Both run inputs remain unchanged. The destination must not overlap either input, and its remaining capacity must hold the full decoded stream.

Parameters:
  • group (ThreadGroup) – Complete one-dimensional block. Every thread must participate.

  • run_values (ThreadData or CuTe register payload) – Matching blocked run payloads, as in cuda.coop.cutlass.run_length_decode().

  • run_lengths (ThreadData or CuTe register payload) – Matching blocked run payloads, as in cuda.coop.cutlass.run_length_decode().

  • destination (cute.Tensor) – Contiguous one-dimensional global-memory tensor with the run-value dtype and a known static or runtime extent. All block members supply the same tensor. Bare pointers lack the capacity needed for validation.

  • decoded_items_per_thread (int) – Positive compile-time number of items per thread in each decode window. Each window holds block_threads * decoded_items_per_thread items; the final window writes only the remaining valid items.

  • destination_offset (integer, optional) – Block-uniform nonnegative output index, default zero. Static and runtime integers up to uint64 are checked before pointer arithmetic.

  • temp_storage (TempStorage, optional) – Scratch covering the complete operation. Synchronize before reuse when its descriptor sets auto_sync=False.

Returns:

Full decoded size, available to every member. Empty input writes nothing. Insufficient capacity traps before any output write; elements outside the decoded interval remain unchanged. The total must fit uint32.

Return type:

cutlass.Uint32

Notes

The full stream is decoded; destination_offset chooses where it is written. Offset and capacity validation complete before any output write. Negative lengths, misplaced zero padding, and total-size overflow follow cuda.coop.cutlass.run_length_decode().

See the Run Length Decode visualization for bulk output and run ordering.

Examples

Write the full decoded stream at destination offset five, alongside a register window of the same runs. Every thread receives the total decoded length.

The launcher accepts device pointers and a compile-time items_per_thread value.

import cutlass
from cutlass import cute

import cuda.coop.cutlass as coop

@cute.kernel
def decode_runs(
    window_pointer: cute.Pointer,
    stream_pointer: cute.Pointer,
    totals_pointer: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    block = coop.this_block()
    values = coop.ThreadData(items_per_thread)
    lengths = coop.ThreadData(items_per_thread)
    for item in cutlass.range_constexpr(items_per_thread):
        values[item] = cutlass.Int32(
            block.rank() * items_per_thread + item + 10
        )
        lengths[item] = cutlass.Uint32(2)
    scratch = coop.TempStorage(auto_sync=True)
    window = coop.run_length_decode(
        block,
        values,
        lengths,
        decoded_items_per_thread=2,
        decoded_window_offset=3,
        temp_storage=scratch,
    )
    coop.store(block, window_pointer, window)
    destination = cute.make_tensor(
        stream_pointer, cute.make_layout(64 * items_per_thread + 5)
    )
    total = coop.run_length_decode_into(
        block,
        values,
        lengths,
        destination,
        decoded_items_per_thread=1,
        destination_offset=5,
        temp_storage=scratch,
    )
    totals = cute.make_tensor(totals_pointer, cute.make_layout(32))
    totals[block.rank()] = total

@cute.jit
def launch(
    window_pointer: cute.Pointer,
    stream_pointer: cute.Pointer,
    totals_pointer: cute.Pointer,
    items_per_thread: cutlass.Constexpr,
):
    decode_runs(
        window_pointer, stream_pointer, totals_pointer, items_per_thread
    ).launch(grid=1, block=32)