cuda.coop.cutlass API Reference#
Warning
This is an experimental API and is subject to change.
Use import cuda.coop.cutlass as coop inside programs compiled with CuTe.
The installed .pyi files describe overload and result typing. The
Programming Guide explains payload ownership,
participation, layouts, and operation-specific controls.
Load always returns a fresh ThreadData, including when
items_per_thread=1. Global and shared tensors can be Load sources or
Store destinations. CuTe register and immutable-vector adapters provide the
conversion rules for operations accepting those forms.
Reduce and Sum return CuTe scalars, even when a NumPy dtype selects the
payload type; stubs report that NumPy type for static checks. Array operations
return ThreadData unless their documented contract says otherwise. Warp
Scan accepts one scalar per lane. Rank results and Discontinuity flags have
signed Int32 dtype. Bulk Run Length Decode returns a Uint32 total to
every block member.
Recognized operator functions and NumPy ufuncs select built-in operators.
For custom Reduce and Scan binary operators, prepare a DeviceOperator
with compile_operator() before tracing. See
Compiled scalar device operators for the scalar signature and compiler
requirements. Scan prefix callbacks and runtime callback state are unsupported.
Compiled device operators#
- cuda.coop.cutlass.compile_operator( ) DeviceOperator#
Compile a stateless
@cute.jitbinary operator before kernel tracing.- Parameters:
func (callable) – A
@cute.jitfunction taking two scalar operands and returning one scalar. Annotate its return type with the requested CUTLASS dtype. The function must implement an associative operation for scan and reduction. Runtime captures and stateful prefix callbacks are unsupported.dtype (type) – A supported CUTLASS or NumPy scalar type, or
int/float.arch (str or None) – CUDA target such as
"sm_90a". None uses the configured CuTe target or the current GPU. Kernels using the result must target the same arch.
- Returns:
Immutable compiled operator accepted by scan and reduction.
- Return type:
- class cuda.coop.cutlass.DeviceOperator( )#
A stateless scalar binary operator prepared by
compile_operator().dtypeis the CUTLASS type of both operands and the return value.archrecords the CUDA target required by kernels using this operator. The compiled device code is retained so a removed cache file can be rebuilt.
Thread groups#
- cuda.coop.cutlass.this_thread() ThreadGroup#
Describe the calling CUDA thread.
- cuda.coop.cutlass.this_warp() ThreadGroup#
Describe the calling physical 32-thread warp.
Construction does not inspect active lanes or synchronize threads. A consuming query or primitive resolves the required launch facts and applies that operation’s participation rules.
- cuda.coop.cutlass.this_block() ThreadGroup#
Describe the current CUDA thread block for a CUTLASS primitive.
- cuda.coop.cutlass.this_cluster() ThreadGroup#
Describe the cluster of the current kernel launch.
The consuming query or primitive resolves launch facts. Constructing this descriptor does not configure cluster scheduling.
- cuda.coop.cutlass.this_grid() ThreadGroup#
Describe the current grid for supported hierarchy queries.
Queries resolve the launch dimensions when consumed. This descriptor does not enable grid synchronization or reductions.
- class cuda.coop.cutlass.ThreadGroup(
- kind: str,
- hierarchy: ThreadHierarchy = <factory>,
- parent: ThreadGroup | None = None,
- mapping: GroupByMapping | None = None,
- source: str = 'explicit',
Describe a group and emit CuTe hierarchy queries on demand.
Construction and group_by create descriptors. Rank, count, membership, and synchronization methods emit device operations when called during tracing. Query results are CuTe scalar values. See thread groups for hierarchy and participation.
- group_by( ) _ThreadGroupT#
Partition a physical warp by threads or a block by warps.
- Parameters:
count (int) – Compile-time positive number of units in each group. The units are threads for a physical-warp parent and physical warps for a block parent. For example,
this_block().group_by(2)describes 64 threads. The count cannot exceed the parent’s unit count.exhaustive (bool, optional) – Compile-time flag, default
True. If true,countmust divide the parent’s unit count exactly. If false, trailing units that cannot form a complete group are excluded; queryis_member()before using their ranks.
- Returns:
A descriptor for the calling thread’s mapped group. Nested
group_by()calls are unsupported.- Return type:
Notes
The enclosing block must contain complete physical warps. Logical Warp primitives support widths of 1, 2, 4, 8, 16, or 32. Mapped groups may query their constituents and immediate physical parent, but not a higher hierarchy level. Groups of physical warps support queries and have no explicit synchronization methods. See thread groups and participation requirements.
- rank(level='thread')#
Query rank with the default unsigned CuTe result type.
The optional
levelselects the queried hierarchy level. Queries involving the grid return Uint64; other queries return Uint32. See thread groups for the hierarchy.
- count(level='thread')#
Query count with the default unsigned CuTe result type.
The optional
levelselects the queried hierarchy level. Queries involving the grid return Uint64; other queries return Uint32. See thread groups for the hierarchy.
- rank_as(dtype=None, level='thread')#
Query rank using an optional integer dtype selector.
The result remains a CuTe scalar even when dtype is a Python or NumPy type. None keeps the default unsigned width.
levelhas the same meaning as incuda.coop.cutlass.ThreadGroup.rank().
- count_as(dtype=None, level='thread')#
Query count using an optional integer dtype selector.
The result remains a CuTe scalar even when dtype is a Python or NumPy type. None keeps the default unsigned width.
levelhas the same meaning as incuda.coop.cutlass.ThreadGroup.count().
- is_member()#
Return a CuTe Uint8 flag for membership in this group.
Check membership before using a mapped rank. Primitive participation rules still apply; a membership guard alone cannot make a divergent collective valid.
- sync()#
Synchronize the participating members of this group.
- sync_aligned()#
Synchronize a converged group with aligned participation.
- class cuda.coop.cutlass.ThreadHierarchy#
CUDA hierarchy descriptor for the current kernel launch.
ThreadHierarchy()andThreadHierarchy.current()describe the active launch.Hierarchyis an alias for this class. The compiler supplies the block, grid, and optional cluster dimensions from the kernel launch; this constructor accepts no dimensions.Use
cuda.coop.cutlass.this_block(),cuda.coop.cutlass.this_warp(), or another group factory in a kernel to obtain acuda.coop.cutlass.ThreadGroup. See thread groups for the hierarchy and ranks and sizes for runtime queries. The corresponding C++ abstraction is cuda::hierarchy.Notes
ThreadHierarchy()andcurrent()leaveblock_dim,grid_dim, andcluster_dimasNone. Creating one does not resolve runtime ranks or allocate device storage. Use group methods inside a kernel to query the actual launch.
- class cuda.coop.cutlass.Hierarchy#
Alias for
ThreadHierarchy.
Per-thread payloads and CuTe conversion#
- class cuda.coop.cutlass.ThreadData(
- items_per_thread: int,
- dtype: Any = None,
- *,
- values: tuple[Any, ...] | list[Any] | None = None,
- alignment: int | None = None,
Hold a fixed number of items per thread inside a CuTe kernel.
Primitives infer the item count from the payload. Indexing uses compile-time integers; initialize each item before reading it or passing it to an operation that reads the payload.
The conversion methods below copy values between this payload and CuTe registers. Use
cuda.coop.cutlass.load()andcuda.coop.cutlass.store()to move data to or from global or shared memory. See CUTLASS register payloads for the distinction between mutable register tensors andTensorSSA.- Parameters:
items_per_thread (int) – Positive compile-time item count.
dtype (type, optional) – Optional numeric element-type metadata. Load supplies the memory element type; consuming primitives can infer it from homogeneous initialized values. Advanced interop may need explicit metadata when a raw integer IR value has no signedness. See Element types in advanced interop. The constructor does not cast entries supplied in
values.values (tuple or list, optional) – Initial values, with exactly
items_per_threadentries. When omitted, the items remain uninitialized until assigned or loaded.alignment (int, optional) – Minimum byte alignment for materialized storage. Must be a positive power of two. The backend may select a larger alignment.
Examples
Convert a CuTe register tensor to a payload, change one item, and export both immutable and mutable register results. The original tensor retains its values:
import cutlass from cutlass import cute import cuda.coop.cutlass as cutlass_coop @cute.kernel def convert_registers(destination: cute.Pointer): thread = cute.arch.thread_idx()[0] original = cute.make_rmem_tensor(2, cutlass.Int32) original[0] = thread * 2 original[1] = thread * 2 + 1 items = cutlass_coop.ThreadData.from_register_tensor(original) items[0] = items[0] + 10 vector = items.to_tensor_ssa() copied = items.to_register_tensor() copied[1] = copied[1] + 100 output = cute.make_tensor(destination, cute.make_layout(64 * 6)) for item in cutlass.range_constexpr(2): output[thread * 6 + item] = original[item] output[thread * 6 + 2 + item] = vector[item] output[thread * 6 + 4 + item] = copied[item] @cute.jit def launch(destination: cute.Pointer): convert_registers(destination).launch(grid=1, block=64)
- classmethod ThreadData.from_values( ) ThreadData#
Construct a payload from one or more per-thread scalar values.
- Parameters:
*values (scalar) – Initial items, in payload order. At least one value is required.
dtype (type, optional) – Dtype metadata. Values are stored as supplied, without casting.
- Returns:
A new payload containing
len(values)initialized items.- Return type:
- classmethod ThreadData.from_fn( ) ThreadData#
Initialize each item with a compile-time Python callable.
- Parameters:
items_per_thread (int) – Positive compile-time item count.
fn (callable) – Called once for each integer index from zero to
items_per_thread - 1while tracing the kernel. It may build CuTe scalar expressions; it is not a device callback.dtype (type, optional) – Dtype for the generated values. An explicit CUTLASS numeric type casts each item to that type.
- Returns:
A new initialized payload in increasing item-index order.
- Return type:
- classmethod ThreadData.from_register_tensor( ) ThreadData#
Copy a CuTe register tensor into a per-thread payload.
- Parameters:
fragment (cute.Tensor) – Tensor in CuTe register memory (
rmem). Global- and shared-memory tensors are rejected; load their data first.items_per_thread (int, optional) – Positive compile-time item count. Inferred from the static tensor shape when available. An explicit count must match the inferred count; it is required when inference is unavailable.
dtype (type, optional) – Defaults to
fragment.element_type. An explicit CUTLASS numeric type casts each extracted item.
- Returns:
A new payload in the tensor’s flat integer-index order. Subsequent assignments to either object do not update the other.
- Return type:
- classmethod ThreadData.from_vector( ) ThreadData#
Copy an immutable register value into a per-thread payload.
- Parameters:
vector (cute.TensorSSA or CUTLASS register vector) – Integer-indexable register value. Memory-backed tensors and arrays are rejected. For mutable CuTe
rmemtensors, usefrom_register_tensor().items_per_thread (int, optional) – Positive compile-time item count, normally inferred from
numel()or a static one-dimensional shape. An explicit count must match any inferred count.dtype (type, optional) – Defaults to the value’s dtype metadata. An explicit CUTLASS numeric type casts each extracted item.
- Returns:
A new payload in flat integer-index order. The input is unchanged.
- Return type:
- classmethod ThreadData.from_payload( ) ThreadData#
Adapt a
ThreadDataor CuTe register payload.- Parameters:
payload (ThreadData, cute.Tensor, cute.TensorSSA, or register vector) – Per-thread values. CuTe tensors must use register memory; global/shared tensors and arrays require a load first.
items_per_thread (int, optional) – Positive compile-time item count. If supplied, it must match the existing or inferred count.
dtype (type, optional) – Dtype for conversion. For an existing
ThreadDatawith dtype metadata, this must match that dtype; this method does not recast an already typed payload.
- Returns:
The same object when
payloadis alreadyThreadDataand no dtype change is needed. Supplying a dtype to an untypedThreadDatacreates a new payload. CuTe inputs produce a new payload throughfrom_register_tensor()orfrom_vector().- Return type:
- classmethod ThreadData.load(
- source: ThreadDataLoadSource,
- *,
- items_per_thread: int | None = None,
- dtype: Any = None,
Load a producer-owned source into per-thread registers.
The source must explicitly implement the private
__cuda_coop_thread_data_load__tracing hook. The producer owns any required tensor-memory-to-register copy, including its copy atom, thread partition, readiness, and lifetime. The hook must return eitherThreadDataor a statically sized per-thread register payload accepted byfrom_payload(). The hook is invoked exactly once by each call during DSL tracing; a single-use producer source is responsible for rejecting a second call.Bare memory-backed tensors are intentionally not interpreted here: their address space and shape do not provide enough information to safely select or schedule a register load.
items_per_threadanddtypeare optional static constraints on the returned register payload, matchingfrom_payload().
- ThreadData.to_tensor_ssa(
- *,
- dtype: Any = None,
- shape: Any = None,
- like: ThreadDataTensorMetadata | None = None,
Export initialized items as an immutable CuTe register value.
- Parameters:
dtype (type, optional) – Supported CUTLASS or NumPy numeric dtype. Defaults to this payload’s dtype; required if the payload has no dtype metadata.
shape (int or tuple, optional) – Defaults to
(items_per_thread,). May be nested, but every extent must be a positive compile-time integer and the total number of elements must equalitems_per_thread.like (ThreadDataTensorMetadata, optional) – Supplies missing shape and dtype metadata. Explicit arguments take precedence. Flat item order is preserved; producer-specific lane ownership is not reconstructed by this metadata.
- Returns:
Register values with the requested dtype and shape, in payload item order. The original input fragment’s shape is not retained automatically. Later payload assignments do not change this value.
- Return type:
cute.TensorSSA
- ThreadData.to_register_tensor( ) Any#
Export initialized items to a new mutable CuTe register tensor.
- Parameters:
dtype (type, optional) – Output dtype, with the same rules as
to_tensor_ssa().shape (int or tuple, optional) – Static output shape, with the same rules as
to_tensor_ssa().
- Returns:
A new
rmemtensor containing the payload values. It does not alias the payload or an earlier input fragment. Its storage honors this payload’s minimum alignment. As with other register tensors, the compiler may spill values to local memory.- Return type:
cute.Tensor
Producer protocols#
- class cuda.coop.cutlass.ThreadDataLoadSource(*args, **kwargs)#
Producer capability that loads one payload into per-thread registers.
Implementations own the storage-specific copy policy and expose it through this trace-time hook.
- class cuda.coop.cutlass.ThreadDataTensorMetadata(*args, **kwargs)#
Static shape and dtype used to reconstruct a CuTe
TensorSSA.
- class cuda.coop.cutlass.ThreadDataSource(*args, **kwargs)#
Producer source that supports loading and TensorSSA reconstruction.
See producer interoperability for capability detection, copy ownership, and lifetime requirements.
Temporary storage#
- class cuda.coop.cutlass.TempStorage(
- size_in_bytes=None,
- *,
- alignment=None,
- auto_sync=False,
- sharing='shared',
Describe shared-memory requirements for operations in one kernel.
Supported block algorithms accept an explicit descriptor.
reserveexposes typed arrays for application and library code. See temporary storage for allocation lifetime and the CUTLASS example for reuse.- size_in_bytes#
Explicit positive capacity, or
Noneto infer it from all uses, including reserved arrays. An explicit capacity must cover the resolved allocation and fit a signed 32-bit integer. Device shared-memory limits still apply.- Type:
int or None
- alignment#
Optional minimum byte alignment, a power of two up to 1024. The planner also satisfies the C++ primitive’s alignment requirement.
- Type:
int or None
- auto_sync#
Whether each scratch-using call emits a trailing block barrier. Defaults to
False;Noneis normalized toFalse. Without automatic barriers, the caller must synchronize before scratch reuse.- Type:
- sharing#
Shared calls and reservations alias one slice sized for the largest requirement. Exclusive calls and reservations get separate slices in trace order. Repeated execution of one call site reuses its slice, including in a loop.
- Type:
{“shared”, “exclusive”}
- Equal settings do not combine separate descriptors. Reuse the same object
- to share an allocation across calls in a kernel. Finalization validates
- descriptor lifetimes and supplies the storage operands. ``release``
- consumes the descriptor and returns its complete region as a native view.
- Runtime branches and loops may carry aliases of the same kernel-local
- descriptor; selecting or rebinding to a different descriptor is rejected.
- Use an ``if`` statement for conditional operations, with a condition uniform
- across participating threads, rather than a runtime conditional expression.
- TempStorage.reserve(num_elems, dtype, *, alignment=None)#
Return a one-dimensional CuTe shared tensor with a fixed extent.
num_elems,dtype, and optional bytealignmentmust be compile-time constants. Supported dtypes are 8/16/32/64-bit signed and unsigned integers, Float16, BFloat16, Float32, and Float64. Ordinary NumPy scalar types and dtypes are accepted where available.The descriptor’s
sharingpolicy applies to both reservations and primitive scratch. Shared views alias the same bytes; exclusive views occupy separate slices. Each traced call site has one view, including when it executes repeatedly in a runtime loop. Separate descriptors always own separate allocations.Reservations require
auto_sync=False. The caller must finish all accesses and asynchronous library operations before reusing overlapping bytes, and synchronize participating threads as needed.sync()is a block barrier, not an asynchronous completion wait.
- TempStorage.release(dtype=None)#
Return the entire existing region as a native shared CuTe tensor.
dtypeis a supported compile-time scalar type; omission returns bytes (Uint8). The resolved capacity includes exclusive slices, padding, and any explicit capacity. Its size must be divisible by the element size and its alignment must satisfy that type. An empty inferred region is rejected. Inferred extents resolve at finalization. The descriptor must be constructed inside the kernel.This consumes every alias of the descriptor on the executing path. Subsequent primitives, reserve, sync, or release calls are compile errors, including across loop iterations. A fresh constructor inside a loop creates a new logical instance. Release adds no allocation, barrier, or asynchronous completion wait; the caller completes prior work before using overlapping bytes through the returned tensor.
Memory operations#
- cuda.coop.cutlass.load(
- group: ThreadGroup,
- source: object,
- /,
- *,
- items_per_thread: int,
- dtype: object | None = None,
- alignment: int | None = None,
- algorithm: Literal['direct', 'striped', 'vectorize', 'transpose', 'warp_transpose', 'warp_transpose_timesliced'] = 'direct',
- valid_items: int | integer | CompilerIntegerLike | None = None,
- oob_default: _NumericT | float | None = None,
- offset: int | integer | CompilerIntegerLike | None = None,
- temp_storage: TempStorageLike | None = None,
Load a contiguous group tile into per-thread values.
Supply a positive compile-time
items_per_threadto return fresh ThreadData with the source dtype. The result owns independent values, including when the count is one. Blocks and physical or logical warps are supported.The source dtype is the element type its CuTe operand reports. CuTe reports integer memory behind a
cute.Pointeras signed, including when the host created the pointer with an unsigned type. The exception is a pointer that the kernel itself made withcute.make_ptrand passes unchanged. Acute.make_tensorview of such memory is signed as well.Uint32data then loads asInt32with no error, and a maximum, a sort, or a widening conversion gives the signed answer.For unsigned memory, pass
dtypewith the unsigned type of the same width, such asdtype=cutlass.Uint32. The result has that dtype, andoob_defaultis checked against it.dtypemay also repeat the reported type. Any otherdtypeis an error, so it cannot change the element width or turn integers into floats. Acute.recast_tensorview states its element type and needs nodtype.alignmentrequests a minimum power-of-two byte alignment for the result’s materialized storage, as inThreadData. It does not describe source memory alignment or change the selected item layout. To update a native CuTe register tensor, usetensor.store(values.to_tensor_ssa())with compatible shape and item order. The Load implementation materializes a temporary register tensor and copies its scalar values into ThreadData.Load populates the payload in the selected algorithm’s layout. Beyond
valid_items, slots have unspecified values unlessoob_defaultis supplied. A default requiresvalid_items. A runtime default must have the memory dtype. A Python or NumPy default must be finite and within the memory dtype’s range; a float default requires a floating-point dtype.The count ranges from zero through the group’s full tile size.
offsetis a nonnegative element offset. Counts, offsets, and supplied defaults must agree within the group. Physical and logical warps receive consecutive tiles in linear block-thread order; the compiler adds that group’s tile origin to the offset. Different groups can use different controls. The caller must provide enough accessible memory for the selected prefix at the resulting offset.For
this_warp().group_by(width),widthmust be 1, 2, 4, 8, 16, or 32. Each logical tile containswidth * items_per_threadelements. The enclosing block must contain complete 32-thread physical warps.The source must expose a raw pointer and a provably compact layout. A bare pointer object with no shape or stride metadata is also accepted, but its capacity is not checked. Register or local-memory tensors are rejected. Load reads addressable memory, such as global or shared memory. Shared tensors from
SmemAllocatorandTempStorage.reserveare supported when their dtype and compact layout meet these requirements. Synchronize the participating threads after shared-memory writes and before Load reads values produced by other threads. DIRECT, STRIPED, and VECTORIZE need no shared scratch or reuse barrier; an accepted block descriptor does not change that.Transpose algorithms use shared scratch. With no
temp_storage, the compiler allocates it and inserts a trailing reuse barrier. An explicitcuda.coop.cutlass.TempStorageis supported only for block calls. It sets size, alignment, sharing, and synchronization policy; its defaultauto_sync=Falserequires a barrier before reuse. Each physical or logical warp uses an independent scratch slice and a reuse barrier that covers only its own lanes.Examples
Copy a partial tile between different source and destination offsets. The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def copy_prefix( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() valid_items = 64 * items_per_thread - 3 values = coop.load( block, source, items_per_thread=items_per_thread, valid_items=valid_items, oob_default=0, offset=5, ) coop.store( block, destination, values, valid_items=valid_items, offset=2, ) @cute.jit def launch( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): copy_prefix(source, destination, items_per_thread).launch( grid=1, block=64 )
See also
- cuda.coop.cutlass.store(
- group: ThreadGroup,
- destination: object,
- value: _NumericT | _ReadableThreadDataLike[_NumericT],
- /,
- *,
- dtype: object | None = None,
- algorithm: Literal['direct', 'striped', 'vectorize', 'transpose', 'warp_transpose', 'warp_transpose_timesliced'] = 'direct',
- valid_items: int | integer | CompilerIntegerLike | None = None,
- offset: int | integer | CompilerIntegerLike | None = None,
- temp_storage: TempStorageLike | None = None,
Store per-thread values into a contiguous group tile.
Blocks and physical or logical warps are supported. Each thread supplies a scalar or an initialized CUTLASS ThreadData payload whose dtype matches the destination. Do not rely on the payload’s contents after a transpose Store; copy values needed later.
dtypedeclares the destination’s element type as it does forload(). To store an unsigned payload through acute.Pointer, which reports the signed type of the same width, pass the payload’s dtype. The declared type must equal the payload dtype.valid_itemsselects a prefix from zero through the group’s full tile size.offsetis a nonnegative element offset. Both must agree within the group. Physical and logical warp widths, tile origins, and prefix bounds followload(). Different groups can use different controls. The caller must provide enough accessible destination memory for the prefix at the resulting offset. Items outside it are not written. The destination has the same pointer and layout requirements asload().Transpose algorithms use shared scratch. With no
temp_storage, the compiler allocates it and inserts a trailing reuse barrier. An explicit descriptor is supported only for block calls and controls allocation and reuse. The caller must synchronize before reuse unlessauto_sync=True. Physical and logical warps use independent scratch slices and a barrier that covers only the group’s lanes. DIRECT, STRIPED, and VECTORIZE need no scratch or reuse barrier.Examples
Copy a partial tile between different source and destination offsets. The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def copy_prefix( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() valid_items = 64 * items_per_thread - 3 values = coop.load( block, source, items_per_thread=items_per_thread, valid_items=valid_items, oob_default=0, offset=5, ) coop.store( block, destination, values, valid_items=valid_items, offset=2, ) @cute.jit def launch( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): copy_prefix(source, destination, items_per_thread).launch( grid=1, block=64 )
See also
Reduction#
- cuda.coop.cutlass.reduce(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | _ItemT | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- binary_op: Literal['+', 'sum', 'add', 'plus', '*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- algorithm: Literal['raking_commutative_only', 'raking', 'warp_reductions'] | None = None,
- temp_storage: TempStorageLike | None = None,
Reduce scalars or register payloads with an associative operator.
Every member of the block or warp participates. Only group rank zero may read the result. CuTe register payloads contribute all their elements.
- Parameters:
group (ThreadGroup) – Block, physical warp, or logical warp. Logical widths may be powers of two from 1 through 32, or widths from 17 through 31. For the latter, use
group_by(width, exhaustive=False)and guard the call withgroup.is_member(). Every member participates, including members excluded byvalid_items. The enclosing block must contain complete physical warps for warp reductions.value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Each member supplies one scalar or a fixed-size per-thread payload. Register tensors and
TensorSSAvalues are converted withcuda.coop.cutlass.ThreadData.from_payload(). Every payload element contributes to one scalar result. Inputs remain unchanged. Supported dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, andFloat64.binary_op (str, built-in alias, or DeviceOperator, optional) – Compile-time operator, default sum. Supported strings are
"sum","multiplies","min","max","bit_and","bit_or", and"bit_xor". Also accepts the correspondingoperatoror NumPy aliases, such asoperator.addandnumpy.maximum. Bitwise operators require an integer dtype. A device operator prepared withcompile_operator()must accept and return the input dtype and be associative. It cannot use"raking_commutative_only".valid_items (integer, optional) – Group-uniform count of contributing members, from one through the group size. Requires scalar input.
Noneincludes every member; empty reductions are unsupported.algorithm (str, optional) – Compile-time block algorithm:
"raking_commutative_only","raking", or"warp_reductions".Noneselects"warp_reductions". Warp groups requireNone.temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Size and alignment are inferred from its uses unless requested. Omission enables automatic allocation and reuse barriers. With
auto_sync=False, synchronize the block before reuse. Warp groups use automatic storage and requireNone.
- Returns:
Reduced value with the input dtype, defined only at group rank zero. A NumPy dtype selector still produces a CuTe scalar inside the kernel.
- Return type:
CuTe numeric scalar
Notes
Every reduction uses CUB. Blocks and warps accept scalars or per-thread payloads. A
valid_itemsprefix counts contributing members, requires scalar input, and does not reduce required participation. Block reductions accepttemp_storagewith the same sharing, capacity, alignment, and synchronization policies as Load/Store. Floating-point results can differ from a sequential fold because reduction regroups operations.See also
- cuda.coop.cutlass.sum
Sum with the same operand and result behavior.
cub::BlockReduce,cub::WarpReduceC++ primitives used for block and warp reductions.
Examples
Compute a sum and a maximum over the same tile. Only the block root reads either result.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def summarize( source: cute.Pointer, totals: cute.Pointer, maxima: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) total = coop.sum(block, values) maximum = coop.reduce(block, values, binary_op="max") total_output = cute.make_tensor(totals, cute.make_layout(1)) maximum_output = cute.make_tensor(maxima, cute.make_layout(1)) if block.rank() == 0: total_output[0] = total maximum_output[0] = maximum @cute.jit def launch( source: cute.Pointer, totals: cute.Pointer, maxima: cute.Pointer, items_per_thread: cutlass.Constexpr, ): summarize(source, totals, maxima, items_per_thread).launch( grid=1, block=64 )
See also
- cuda.coop.cutlass.reduce_batched(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- binary_op: Literal['+', 'sum', 'add', 'plus', '*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | Callable[[object, object], object] | None = None,
- output_layout: Literal['striped', 'blocked'] = 'striped',
Reduce each payload slot independently across a warp.
Each slot is an independent batch: slot
bin every lane contributes to batchb. The input is preserved. ReadableThreadData, CuTe register tensors, andTensorSSAvalues are accepted throughThreadData.from_payload.binary_opaccepts the built-in strings and aliases documented bycuda.coop.cutlass.reduce(). Custom callbacks are not supported.output_layoutis"striped"or"blocked". ForBbatches andWlanes, the returnedThreadDatacontainsceil(B / W)slots per lane. Only slots corresponding to a batch are defined. Guard reads using the batch index.The CUB provider exchanges register values within the participating warp. It requires no shared scratch allocation or storage-reuse barrier. Logical warps may call it independently, including from different branches.
- Parameters:
group (ThreadGroup) – Complete physical warp or logical warp of 1, 2, 4, 8, 16, or 32 lanes. All members of the selected group participate.
value (ThreadData or CuTe register payload) – One input per independent batch in each lane. All lanes use the same positive batch count and dtype: signed or unsigned 8-, 16-, 32-, or 64-bit integers,
Float32, orFloat64. The input is preserved.binary_op (str or recognized built-in alias, optional) – Reduction operator, default addition. Bitwise operators require integer inputs. The operator must be associative and commutative. Custom callbacks are not supported.
output_layout ({"striped", "blocked"}, optional) – Result ownership, default striped. Striped slot
iin lanetholds batcht + i * W; blocked slotiholds batcht * ceil(B / W) + i. Read only indices belowB.
- Returns:
Fresh payload with the input dtype and
ceil(B / W)slots per lane. Accumulation uses the input dtype without promotion. Slots beyond the batch count are undefined.- Return type:
Notes
See the Batched Warp Reduction visualization for lane and result ownership.
Examples
Sum each per-thread slot independently within two physical warps. Guard result reads by the batch index: lanes without a batch have no defined result.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def sum_columns( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() warp = coop.this_warp() values = coop.load(block, source, items_per_thread=items_per_thread) totals = coop.reduce_batched(warp, values) output = cute.make_tensor( destination, cute.make_layout(2 * items_per_thread) ) for item in cutlass.range_constexpr((items_per_thread + 31) // 32): batch = warp.rank() + item * 32 if batch < items_per_thread: index = block.rank() // 32 * items_per_thread + batch output[index] = totals[item] @cute.jit def launch( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): sum_columns(source, destination, items_per_thread).launch( grid=1, block=64 )
See also
- cuda.coop.cutlass.sum(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | _ItemT | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- algorithm: Literal['raking_commutative_only', 'raking', 'warp_reductions'] | None = None,
- temp_storage: TempStorageLike | None = None,
Sum scalars or per-thread register payloads with CUTLASS.
Equivalent to
cuda.coop.cutlass.reduce()withbinary_op="sum". Its group, payload, valid-prefix, algorithm, and scratch contracts apply. Every group member participates; only rank zero may read the result.- Parameters:
group (ThreadGroup) – Block, physical warp, or logical warp supported by
reduce.value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Each member’s contribution. Every payload item contributes; the input remains unchanged.
valid_items (integer, optional) – Uniform count of contributing members, for scalar inputs only.
algorithm (str, optional) – Block algorithm described by
reduce; omit for warp groups.temp_storage (TempStorage, optional) – Explicit block scratch descriptor; omit for warp groups. Omission enables automatic scratch and reuse synchronization.
- Returns:
Sum with the input dtype, defined only at group rank zero.
- Return type:
CuTe numeric scalar
See also
- cuda.coop.cutlass.reduce
Built-in operators, register payloads, and result ownership.
cub::BlockReduce,cub::WarpReduceC++ primitives used for block and warp reductions.
Examples
Write a tile sum and maximum from the block root.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def summarize( source: cute.Pointer, totals: cute.Pointer, maxima: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) total = coop.sum(block, values) maximum = coop.reduce(block, values, binary_op="max") total_output = cute.make_tensor(totals, cute.make_layout(1)) maximum_output = cute.make_tensor(maxima, cute.make_layout(1)) if block.rank() == 0: total_output[0] = total maximum_output[0] = maximum @cute.jit def launch( source: cute.Pointer, totals: cute.Pointer, maxima: cute.Pointer, items_per_thread: cutlass.Constexpr, ): summarize(source, totals, maxima, items_per_thread).launch( grid=1, block=64 )
See also
Scan#
- cuda.coop.cutlass.scan(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
- /,
- *,
- mode: Literal['exclusive', 'inclusive'] = 'exclusive',
- scan_op: Literal['+', 'sum', 'add', 'plus'] | Literal['*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
- initial_value: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
- temp_storage: TempStorageLike | None = None,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- aggregate_output: ThreadDataLike[Any] | None = None,
Scan register values with optional valid-prefix and aggregate controls.
Prefixes follow linear group rank. Block payloads use blocked order: all items from thread zero, then all items from thread one, and so on. Prefixes restart for each group; this call does not scan across blocks.
- Parameters:
group (ThreadGroup) – Complete block, physical warp, or supported logical warp. Every group member executes the same call. Warp scans require an enclosing block size divisible by 32.
value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Blocks accept scalars or fixed-size per-thread payloads in blocked order. Register tensors and
TensorSSAvalues are converted withcuda.coop.cutlass.ThreadData.from_payload(). All members must use the same dtype and extent. Physical and logical warps accept scalars only, including when each thread would otherwise hold one item.mode ({"exclusive", "inclusive"}, optional) – Compile-time choice, default
"exclusive". Exclusive prefixes combineinitial_valuewith preceding inputs. Inclusive prefixes also include the current item and require no initial value.scan_op (str, built-in alias, or DeviceOperator, optional) – Compile-time operator, default sum. Supports
"sum","multiplies","min","max","bit_and","bit_or", and"bit_xor", and correspondingoperatoror NumPy aliases such asoperator.addandnumpy.maximum. Bitwise operators require an integer dtype. A device operator prepared withcompile_operator()must accept and return the input dtype and be associative. Exclusive mode requires an explicit initial value.initial_value (numeric scalar, optional) – Starting value for exclusive mode. Sum defaults to zero; other operators require an explicit value. A typed CuTe or NumPy scalar must match the input dtype. Python literals must be finite and representable in that dtype. Inclusive mode requires
None. The initial value must be uniform across the group.valid_items (int or CuTe integer scalar, optional) – Warp-only count of contributing lanes, from one through the group size, uniform within the group.
Noneincludes every lane. All lanes participate, but only ranks below this count have defined scan results. The enclosing block must contain complete physical warps. Blocks requireNone.aggregate_output (ThreadData, optional) – Writable one-item payload receiving the input aggregate on every member. Its dtype must match the input, or may be omitted for inference. The aggregate excludes
initial_valueand lanes beyondvalid_items. CuTe register tensors andTensorSSAvalues are not accepted as this output argument.Noneomits the aggregate. Inside a runtime branch or loop, create this output in that region and explicitly copy its item to carry the value beyond the region.algorithm (str, optional) – Compile-time block algorithm:
"raking"(the default),"raking_memoize", or"warp_scans". The last requires a block size divisible by 32. Warp groups requireNone.temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Omission enables automatic scratch and reuse synchronization. Warp groups require
Noneand use automatic storage for each group.
- Returns:
This thread’s prefixes with the input dtype. Scalar input returns a CuTe scalar; payload input returns a new writable
ThreadDatawith the input extent and alignment. The input remains unchanged. Withvalid_items, read only valid lanes.aggregate_outputis written separately, including on lanes outside the valid prefix.- Return type:
CuTe numeric scalar or cuda.coop.cutlass.ThreadData
Notes
Only blocks accept
algorithmand explicittemp_storage. The compiler synchronizes automatically when it manages scratch. An explicitcuda.coop.cutlass.TempStoragedefaults toauto_sync=False. It requires block barriers before reuse; requestauto_sync=Truefor automatic reuse synchronization. Prefix callbacks and callback state are not supported. Scan requires an associative operator and may regroup arithmetic. Floating-point prefixes can differ from a sequential fold; integer arithmetic uses the input dtype.See also
- cuda.coop.cutlass.exclusive_scan
Executable example of a partial logical warp and aggregate output.
cub::BlockScan,cub::WarpScanC++ scan, sum, and aggregate overloads.
Examples
Compare seeded exclusive sums, inclusive maxima, and the sum helpers. One automatically synchronized scratch descriptor serves all four calls.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def prefixes( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) scratch = coop.TempStorage(auto_sync=True) seeded_sums = coop.scan( block, values, initial_value=7, temp_storage=scratch ) running_maxima = coop.inclusive_scan( block, values, scan_op="max", temp_storage=scratch ) before = coop.exclusive_sum(block, values, temp_storage=scratch) through = coop.inclusive_sum(block, values, temp_storage=scratch) coop.store(block, seeded, seeded_sums) coop.store(block, maxima, running_maxima) coop.store(block, exclusive, before) coop.store(block, inclusive, through) @cute.jit def launch( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): prefixes( source, seeded, maxima, exclusive, inclusive, items_per_thread ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.exclusive_sum(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
- /,
- *,
- algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
- temp_storage: TempStorageLike | None = None,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- aggregate_output: ThreadDataLike[Any] | None = None,
Return exclusive sums starting from zero.
Uses the parameters and return behavior of
cuda.coop.cutlass.scan(), with exclusive mode and sum fixed. Usecuda.coop.cutlass.exclusive_scan()for an explicitinitial_valueor a different associative operator. Thevalid_itemsandaggregate_outputcontrols are available here too.- Returns:
Sum of the items preceding each input item, with the input dtype. Payload input produces a fresh payload and remains unchanged. Read only valid lanes when
valid_itemsis supplied.- Return type:
CuTe numeric scalar or cuda.coop.cutlass.ThreadData
See also
cuda.coop.cutlass.exclusive_scanPartial-warp example. For an exclusive sum starting at zero, replace the call with
exclusive_sumand omitscan_opandinitial_value.
Examples
Compute prefixes before each item and compare them with inclusive sums and an exclusive scan starting from seven.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def prefixes( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) scratch = coop.TempStorage(auto_sync=True) seeded_sums = coop.scan( block, values, initial_value=7, temp_storage=scratch ) running_maxima = coop.inclusive_scan( block, values, scan_op="max", temp_storage=scratch ) before = coop.exclusive_sum(block, values, temp_storage=scratch) through = coop.inclusive_sum(block, values, temp_storage=scratch) coop.store(block, seeded, seeded_sums) coop.store(block, maxima, running_maxima) coop.store(block, exclusive, before) coop.store(block, inclusive, through) @cute.jit def launch( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): prefixes( source, seeded, maxima, exclusive, inclusive, items_per_thread ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.inclusive_sum(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
- /,
- *,
- algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
- temp_storage: TempStorageLike | None = None,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- aggregate_output: ThreadDataLike[Any] | None = None,
Return inclusive sums, including each current item.
Uses the parameters and return behavior of
cuda.coop.cutlass.scan(), with inclusive mode and sum fixed. Thevalid_itemsandaggregate_outputcontrols follow that function.- Returns:
Sum through each input item, with the input dtype. Payload input produces a fresh payload and remains unchanged. Read only valid lanes when
valid_itemsis supplied.- Return type:
CuTe numeric scalar or cuda.coop.cutlass.ThreadData
See also
cuda.coop.cutlass.inclusive_scanInclusive prefixes with other associative operators.
cuda.coop.cutlass.exclusive_scanPartial-warp example. For an inclusive sum, replace the call with
inclusive_sumand omitscan_opandinitial_value.
Examples
Compute prefixes through each item alongside exclusive sums and running maxima.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def prefixes( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) scratch = coop.TempStorage(auto_sync=True) seeded_sums = coop.scan( block, values, initial_value=7, temp_storage=scratch ) running_maxima = coop.inclusive_scan( block, values, scan_op="max", temp_storage=scratch ) before = coop.exclusive_sum(block, values, temp_storage=scratch) through = coop.inclusive_sum(block, values, temp_storage=scratch) coop.store(block, seeded, seeded_sums) coop.store(block, maxima, running_maxima) coop.store(block, exclusive, before) coop.store(block, inclusive, through) @cute.jit def launch( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): prefixes( source, seeded, maxima, exclusive, inclusive, items_per_thread ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.exclusive_scan(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
- /,
- *,
- scan_op: Literal['+', 'sum', 'add', 'plus'] | Literal['*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
- initial_value: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
- temp_storage: TempStorageLike | None = None,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- aggregate_output: ThreadDataLike[Any] | None = None,
Return exclusive prefixes with an optional initial value and aggregate.
Uses the parameters and return behavior of
cuda.coop.cutlass.scan(), with exclusive mode fixed. Sum starts from zero unlessinitial_valueis supplied; other operators require it.aggregate_outputreceives the input aggregate without that initial value.- Returns:
Prefix before each input item, with the input dtype. Payload input produces a fresh payload. For a partial warp, only ranks below
valid_itemshave defined prefixes.- Return type:
CuTe numeric scalar or cuda.coop.cutlass.ThreadData
Examples
Scan five lanes of each eight-lane logical warp, starting from seven. Every lane participates and receives the aggregate. Only valid lanes write their prefixes.
operator.addselects the built-in sum.import operator from cutlass import cute import cuda.coop.cutlass as cutlass_coop @cute.kernel def scan_prefix( source: cute.Pointer, destination: cute.Pointer, aggregates: cute.Pointer, ): thread = cutlass_coop.this_block().rank() group = cutlass_coop.this_warp().group_by(8) inputs = cute.make_tensor(source, cute.make_layout(64)) outputs = cute.make_tensor(destination, cute.make_layout(64)) totals = cute.make_tensor(aggregates, cute.make_layout(64)) aggregate = cutlass_coop.ThreadData(items_per_thread=1) result = cutlass_coop.exclusive_scan( group, inputs[thread], scan_op=operator.add, initial_value=7, valid_items=5, aggregate_output=aggregate, ) if group.rank() < 5: outputs[thread] = result totals[thread] = aggregate[0] @cute.jit def launch( source: cute.Pointer, destination: cute.Pointer, aggregates: cute.Pointer, ): scan_prefix(source, destination, aggregates).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.inclusive_scan(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
- /,
- *,
- scan_op: Literal['+', 'sum', 'add', 'plus'] | Literal['*', 'mul', 'multiply', 'multiplies', 'min', 'minimum', 'max', 'maximum', '&', 'bit_and', '|', 'bit_or', '^', 'bit_xor'] | DeviceOperator | Callable[[object, object], object] | None = None,
- algorithm: Literal['raking', 'raking_memoize', 'warp_scans'] | None = None,
- temp_storage: TempStorageLike | None = None,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- aggregate_output: ThreadDataLike[Any] | None = None,
Return inclusive prefixes with an associative operator.
Uses the parameters and return behavior of
cuda.coop.cutlass.scan(), with inclusive mode fixed. Each prefix includes its current item. This function has noinitial_valueparameter. Thevalid_itemsandaggregate_outputcontrols followcuda.coop.cutlass.scan().- Returns:
Inclusive prefixes with the input dtype. Payload input produces a fresh payload and remains unchanged. Read only valid lanes when
valid_itemsis supplied.- Return type:
CuTe numeric scalar or cuda.coop.cutlass.ThreadData
See also
cuda.coop.cutlass.exclusive_scanPartial-warp example. To include each current item, replace the call with
inclusive_scanand removeinitial_value.
Examples
Compute running maxima alongside seeded and unseeded prefix sums.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def prefixes( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) scratch = coop.TempStorage(auto_sync=True) seeded_sums = coop.scan( block, values, initial_value=7, temp_storage=scratch ) running_maxima = coop.inclusive_scan( block, values, scan_op="max", temp_storage=scratch ) before = coop.exclusive_sum(block, values, temp_storage=scratch) through = coop.inclusive_sum(block, values, temp_storage=scratch) coop.store(block, seeded, seeded_sums) coop.store(block, maxima, running_maxima) coop.store(block, exclusive, before) coop.store(block, inclusive, through) @cute.jit def launch( source: cute.Pointer, seeded: cute.Pointer, maxima: cute.Pointer, exclusive: cute.Pointer, inclusive: cute.Pointer, items_per_thread: cutlass.Constexpr, ): prefixes( source, seeded, maxima, exclusive, inclusive, items_per_thread ).launch(grid=1, block=64)
See also
Data rearrangement#
- cuda.coop.cutlass.exchange(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- mode: Literal['striped_to_blocked', 'blocked_to_striped', 'warp_striped_to_blocked', 'blocked_to_warp_striped', 'scatter_to_blocked', 'scatter_to_striped', 'scatter_to_striped_guarded', 'scatter_to_striped_flagged'] = 'striped_to_blocked',
- ranks: _ReadableThreadDataLike[int | signedinteger[Any] | CompilerIntegerLike] | CutlassTensorSample | CutlassTensorSSASample | None = None,
- valid_flags: _ReadableThreadDataLike[int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any]] | CutlassTensorSample | CutlassTensorSSASample | None = None,
- warp_time_slicing: bool = False,
Exchange register payloads, including ranked block scatters.
Every group member must call, including members whose scatter items are all invalid. For
Gthreads andKitems per thread, blocked order assigns logical elementrank * K + itemto a slot; striped order assignsitem * G + rank. Exchange rearranges registers without loading or storing a memory tile.- Parameters:
group (ThreadGroup) – Complete block, physical warp, or logical warp with a power-of-two width from 1 through 32. Warp operations require an enclosing block size divisible by 32.
value (ThreadData, CuTe register tensor, or TensorSSA) – Fixed-size per-thread payload. Register tensors and
TensorSSAvalues are converted withcuda.coop.cutlass.ThreadData.from_payload(). Every member must supply the same dtype and extent. Supported dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, andFloat64. Scalar inputs are unsupported.mode (str, optional) – Compile-time conversion, default
"striped_to_blocked". All groups also support"blocked_to_striped". Blocks additionally support"blocked_to_warp_striped"and"warp_striped_to_blocked", which stripe within each physical warp and require a block size divisible by 32. Block scatter modes are"scatter_to_blocked","scatter_to_striped","scatter_to_striped_guarded", and"scatter_to_striped_flagged". Physical and logical warps support only"striped_to_blocked"and"blocked_to_striped"; their enclosing block must contain complete physical warps.ranks (ThreadData, CuTe register tensor, or TensorSSA, optional) – Destination ranks, required for every scatter mode and rejected for layout conversions. Must have a signed integer dtype and the same per-thread extent as
value. Valid ranks must be distinct across the block and lie in[0, block_tile_size). Guarded scatter skips negative ranks. Other scatter modes require each participating item’s rank to be in range.Noneis the default.valid_flags (ThreadData, CuTe register tensor, or TensorSSA, optional) – Per-item flags, required for
"scatter_to_striped_flagged"and rejected for other modes. Must have an integer, non-boolean dtype and the same extent asvalue. Only items with nonzero flags scatter; their ranks must be distinct and in range.Noneis the default.warp_time_slicing (bool, optional) – Compile-time option, default
False. Reuse block exchange scratch across warps to reduce shared memory usage at the cost of additional synchronization. Requires a block; guarded and flagged scatter do not support it.
- Returns:
New writable payload with the input dtype, extent, and requested alignment in the output layout. All inputs, ranks, and flags remain unchanged. A scatter destination with no valid input is undefined; initialize it before reading it.
- Return type:
Notes
The operation does not check rank uniqueness or bounds. Guarded scatter skips only negative ranks, and flagged scatter skips only items with a zero flag. Duplicate ranks on written items give undefined results. An out-of-range rank on a written item can write past the exchange scratch.
Scratch allocation and trailing synchronization are automatic. See Scratch allocation and reuse for storage reuse and Groups and tiles for participation requirements.
See also
cub::BlockExchange,cub::WarpExchangeC++ exchange primitives.
Examples
Reverse a 128-item block tile by assigning each input its destination rank. Each of the 64 threads holds two items.
import cutlass from cutlass import cute import cuda.coop.cutlass as cutlass_coop @cute.kernel def reverse_tile( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = cutlass_coop.this_block() thread = block.rank() items = cutlass_coop.load( block, source, items_per_thread=items_per_thread ) ranks = cutlass_coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): ranks[item] = cutlass.Int32( 64 * items_per_thread - 1 - (thread * items_per_thread + item) ) reversed_items = cutlass_coop.exchange( block, items, mode="scatter_to_blocked", ranks=ranks ) cutlass_coop.store(block, destination, reversed_items) @cute.jit def launch( source: cute.Pointer, destination: cute.Pointer, items_per_thread: cutlass.Constexpr, ): reverse_tile(source, destination, items_per_thread).launch( grid=1, block=64 )
See also
- cuda.coop.cutlass.shuffle(
- group: ThreadGroup,
- value: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample | _ItemT,
- /,
- *,
- mode: Literal['down', 'up', 'offset', 'rotate'] = 'down',
- distance: Literal[1] | int | integer | CompilerIntegerLike = 1,
Shift register payloads or select another block member’s scalar.
Payload shifts follow flattened blocked order: each thread’s items are consecutive, and a shift crosses thread boundaries as needed. Scalar offset and rotate calls select one other member’s value.
- Parameters:
group (ThreadGroup) – Complete block whose members all call the primitive. Warp, mapped groups, clusters, and grids are unsupported.
value (numeric scalar, ThreadData, CuTe register tensor, or TensorSSA) – Fixed-size payloads may be
ThreadData, CuTe register tensors, orTensorSSAvalues. The latter two are converted withcuda.coop.cutlass.ThreadData.from_payload(). For scalar offset or rotate modes, each thread supplies one numeric scalar. All members use the same dtype and extent. Supported dtypes are signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, andFloat64.mode (str, optional) – Compile-time mode, default
"down". Payloads support only"up"and"down"; the first or last item of the block tile, respectively, is undefined. Up moves inputi - 1to outputi; down moves inputi + 1to outputi. Scalars require"offset"or"rotate": rankrreceives the value from rankr + distance. Offset results are undefined when that source is outside the block; rotate wraps the source rank modulo the block size.distance (int or CuTe integer scalar, optional) – Default
1. Payload shifts require a compile-time unit distance. Scalar modes accept compile-time or runtime distances, which may differ between threads. Offset distances must fit a signed 32-bit integer and may be zero or negative. Rotate requires at least two threads and1 <= distance < block_size. A runtime distance may have a signed integer dtype up to 64 bits or an unsigned integer dtype up to 32 bits, but its value must remain in the mode’s allowed range.
- Returns:
Scalar input produces a CuTe scalar with the input dtype. Payload input produces a new writable
ThreadDatawith the input dtype, extent, and requested alignment. The input remains unchanged. Observe the chosen mode’s boundary rules before reading the result.- Return type:
CuTe numeric scalar or cuda.coop.cutlass.ThreadData
Notes
Scratch allocation and trailing synchronization are automatic. Every block member must call, including threads with out-of-range offset sources. Warp and logical-warp groups are not supported.
See also
cub::BlockShuffleC++ shift, offset, and rotate primitive.
Examples
Rotate one scalar per thread by seven positions in a 64-thread block, including wraparound.
from cutlass import cute import cuda.coop.cutlass as cutlass_coop @cute.kernel def rotate_tile(source: cute.Pointer, destination: cute.Pointer): block = cutlass_coop.this_block() thread = block.rank() inputs = cute.make_tensor(source, cute.make_layout(64)) outputs = cute.make_tensor(destination, cute.make_layout(64)) outputs[thread] = cutlass_coop.shuffle( block, inputs[thread], mode="rotate", distance=7 ) @cute.jit def launch(source: cute.Pointer, destination: cute.Pointer): rotate_tile(source, destination).launch(grid=1, block=64)
See also
Comparison sorting#
- cuda.coop.cutlass.merge_sort_keys(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- descending: bool = False,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- oob_default: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- temp_storage: TempStorageLike | None = None,
Sort keys across a block or warp without changing the input.
Accepts readable ThreadData and CuTe register payloads. Each thread contributes the same fixed number of items in blocked order, and all members of the group must participate.
- Parameters:
group (ThreadGroup) –
this_block(),this_warp(), orthis_warp().group_by(width). Blocks require a power-of-two total thread count and may be multidimensional. Logical warp widths are 1, 2, 4, 8, 16, and 32; their enclosing physical warps must be complete.keys (ThreadData or CuTe register payload) – Per-thread keys in blocked order, with a positive, compile-time extent. Read-only payloads are accepted. CuTe register-memory tensors and
TensorSSAvalues are converted throughThreadData.from_payload. Supported element types are signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, andFloat64.descending (bool, optional) – Compile-time order selector. The default is ascending order. Custom comparison predicates are not supported.
valid_items (integer, optional) – Group-uniform count in
[0, group_size * items_per_thread]. Only this blocked input prefix participates in the sort. Supplyoob_defaulttogether with this argument for a partial tile. Runtime counts accept signed integer types up to 64 bits or unsigned integer types up to 32 bits. Invalid static counts fail compilation; invalid runtime counts trap before conversion to CUB’s count type.oob_default (scalar, optional) – Group-uniform sentinel that sorts after valid keys: an upper bound for ascending order or a lower bound for descending order. A runtime CuTe scalar must match the key type exactly. Representable Python numeric literals are converted to the key type.
temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Warp groups always use automatic storage with independent slices per group. If the descriptor has
auto_sync=False, synchronize the block before reusing its storage.
- Returns:
Fresh sorted keys with the input element type and per-thread extent, including when the input is a CuTe register payload. Output is blocked: thread
towns sorted positionst * items_per_thread + i. Only the firstvalid_itemspositions are defined for a partial tile. Equal keys have no stability guarantee.- Return type:
See also
Examples
Sort a partial tile in ascending order and its key/index pairs in descending order. Stores write only the valid output prefix.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def sort_prefix( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() valid_items = 64 * items_per_thread - 3 keys = coop.load(block, source, items_per_thread=items_per_thread) indices = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): indices[item] = cutlass.Int32( block.rank() * items_per_thread + item ) scratch = coop.TempStorage(auto_sync=True) ordered = coop.merge_sort_keys( block, keys, valid_items=valid_items, oob_default=1000, temp_storage=scratch, ) pair_keys, pair_indices = coop.merge_sort_pairs( block, keys, indices, descending=True, valid_items=valid_items, oob_default=-1000, temp_storage=scratch, ) coop.store(block, ascending, ordered, valid_items=valid_items) coop.store(block, descending, pair_keys, valid_items=valid_items) coop.store(block, positions, pair_indices, valid_items=valid_items) @cute.jit def launch( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): sort_prefix( source, ascending, descending, positions, items_per_thread ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.merge_sort_pairs(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
- values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- descending: bool = False,
- valid_items: int | integer | CompilerIntegerLike | None = None,
- oob_default: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- temp_storage: TempStorageLike | None = None,
Sort keys and their associated values without changing either input.
Accepts readable ThreadData and CuTe register payloads for either operand. Group participation and ordering follow
cuda.coop.cutlass.merge_sort_keys().- Parameters:
group (ThreadGroup) – Complete block, physical warp, or supported logical warp. Block thread counts must be powers of two; enclosing physical warps must be complete for warp operations.
keys (ThreadData or CuTe register payload) – Fixed-size per-thread payloads with equal extents, in blocked order. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers,
Float32, andFloat64are supported.values (ThreadData or CuTe register payload) – Fixed-size per-thread payloads with equal extents, in blocked order. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers,
Float32, andFloat64are supported.descending (bool, optional) – Compile-time order selector. The default is ascending key order. Custom comparison predicates are not supported.
valid_items (integer, optional) – Group-uniform length of the valid blocked input prefix, from zero through the group tile size. Supply it together with
oob_default. Count types and runtime checks matchmerge_sort_keys.oob_default (scalar, optional) – Key sentinel that sorts after valid keys. Runtime CuTe scalars must have the key type; representable Python literals are converted to it.
temp_storage (TempStorage, optional) – Explicit scratch for block groups only. Warp storage is automatic. Retain automatic synchronization or synchronize before scratch reuse.
- Returns:
Fresh sorted keys and corresponding values, each retaining its own input element type and extent. CuTe register inputs also return
ThreadData. Only the valid blocked prefix is defined for a partial tile. Key/value associations are preserved; equal keys have no stability guarantee.- Return type:
See also
Examples
Carry original indices through a descending partial-tile sort. The ascending keys-only call uses the same automatically synchronized scratch descriptor.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def sort_prefix( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() valid_items = 64 * items_per_thread - 3 keys = coop.load(block, source, items_per_thread=items_per_thread) indices = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): indices[item] = cutlass.Int32( block.rank() * items_per_thread + item ) scratch = coop.TempStorage(auto_sync=True) ordered = coop.merge_sort_keys( block, keys, valid_items=valid_items, oob_default=1000, temp_storage=scratch, ) pair_keys, pair_indices = coop.merge_sort_pairs( block, keys, indices, descending=True, valid_items=valid_items, oob_default=-1000, temp_storage=scratch, ) coop.store(block, ascending, ordered, valid_items=valid_items) coop.store(block, descending, pair_keys, valid_items=valid_items) coop.store(block, positions, pair_indices, valid_items=valid_items) @cute.jit def launch( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): sort_prefix( source, ascending, descending, positions, items_per_thread ).launch(grid=1, block=64)
See also
Radix sorting and ranking#
- cuda.coop.cutlass.radix_sort_keys(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample | _KeyT,
- /,
- *,
- begin_bit: int | integer | CompilerIntegerLike = 0,
- end_bit: int | integer | CompilerIntegerLike | None = None,
- descending: bool = False,
- temp_storage: TempStorageLike | None = None,
- blocked_to_striped: bool = False,
Return stable radix-sorted keys without changing the input.
Accepts scalars and readable or CuTe register payloads, including floating-point keys. Output may use blocked or striped ownership. Every block thread participates with the same controls and per-thread extent.
- Parameters:
group (ThreadGroup) – Complete physical block from
this_block(). Multidimensional blocks are supported. The block tile must contain at most 65,535 items.keys (scalar, ThreadData, or CuTe register payload) – Keys in blocked order, with element type
Int32,Uint32,Int64,Uint64,Float32, orFloat64. A scalar contributes one key per thread. Register-memory tensors andTensorSSAvalues are converted throughThreadData.from_payload; read-only payloads are also accepted.begin_bit (integer, optional) – Block-uniform half-open interval in CUB’s transformed key bits, with
0 <= begin_bit < end_bit <= key_width. Begin defaults to zero; omitted end selects the key width. Either bound may be a runtime CuTe integer, signed up to 64 bits or unsigned up to 32 bits. Invalid runtime bounds trap before narrowing to CUB’s integer arguments.end_bit (integer, optional) – Block-uniform half-open interval in CUB’s transformed key bits, with
0 <= begin_bit < end_bit <= key_width. Begin defaults to zero; omitted end selects the key width. Either bound may be a runtime CuTe integer, signed up to 64 bits or unsigned up to 32 bits. Invalid runtime bounds trap before narrowing to CUB’s integer arguments.descending (bool, optional) – Compile-time selector for descending order. The default is ascending.
temp_storage (TempStorage, optional) – Explicit block scratch, otherwise allocated automatically. Requested alignment is a minimum. With
auto_sync=False, synchronize the block before reusing the storage.blocked_to_striped (bool, optional) – Compile-time output-layout selector. The default is blocked output, where thread
towns sorted indicest * items_per_thread + i.Truereturns striped output, with indicesi * block_threads + t. Use a matching striped Store to write those registers in sorted order.
- Returns:
Sorted keys with the input element type. Scalar inputs return a CuTe scalar; payload inputs return fresh
ThreadDataof the same extent, including when the input uses a CuTe register representation. Equal selected digits retain blocked input order in either direction.- Return type:
CuTe scalar or ThreadData
Notes
CUB transforms keys before selecting the requested bits. Signed integers invert their sign bit. Floating-point keys invert all bits when negative and only the sign bit otherwise. Returned keys keep their original bits. Positive and negative zero compare equally; NaNs follow the transformed bit order, with no guarantee of numeric ordering.
Examples
Sort all key bits in ascending order, then sort key/index pairs by their low four bits in descending order. The latter produces striped registers, so its stores use
algorithm="striped".The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def sort_digits( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() keys = coop.load(block, source, items_per_thread=items_per_thread) indices = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): indices[item] = cutlass.Int32( block.rank() * items_per_thread + item ) ordered = coop.radix_sort_keys(block, keys) pair_keys, pair_indices = coop.radix_sort_pairs( block, keys, indices, begin_bit=0, end_bit=4, descending=True, blocked_to_striped=True, ) coop.store(block, ascending, ordered) coop.store(block, descending, pair_keys, algorithm="striped") coop.store(block, positions, pair_indices, algorithm="striped") @cute.jit def launch( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): sort_digits( source, ascending, descending, positions, items_per_thread ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.radix_sort_pairs(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample | _KeyT,
- values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample | _ValueT,
- /,
- *,
- begin_bit: int | integer | CompilerIntegerLike = 0,
- end_bit: int | integer | CompilerIntegerLike | None = None,
- descending: bool = False,
- temp_storage: TempStorageLike | None = None,
- blocked_to_striped: bool = False,
Return stable radix-sorted keys and their associated values.
Accepts scalars and readable or CuTe register payloads, including floating-point keys. Output may use blocked or striped ownership. Neither input is modified. Key transformations and stable tie handling follow
cuda.coop.cutlass.radix_sort_keys().- Parameters:
group (ThreadGroup) – Complete physical block. Every thread participates with the same controls; the block tile contains at most 65,535 items.
keys (scalar, ThreadData, or CuTe register payload) – Matching scalar or fixed-size payload shapes. Payloads have equal per-thread extents and use blocked input order. Key types are
Int32,Uint32,Int64,Uint64,Float32, andFloat64. Values may independently use signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, orFloat64. Read-only and CuTe register payloads are accepted; one operand may beThreadDatawhile the other is a register payload of the same extent.values (scalar, ThreadData, or CuTe register payload) – Matching scalar or fixed-size payload shapes. Payloads have equal per-thread extents and use blocked input order. Key types are
Int32,Uint32,Int64,Uint64,Float32, andFloat64. Values may independently use signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, orFloat64. Read-only and CuTe register payloads are accepted; one operand may beThreadDatawhile the other is a register payload of the same extent.begin_bit (integer, optional) – Block-uniform half-open interval in transformed key bits. Begin defaults to zero and end to the key width. Runtime integer bounds and validation follow
radix_sort_keys.end_bit (integer, optional) – Block-uniform half-open interval in transformed key bits. Begin defaults to zero and end to the key width. Runtime integer bounds and validation follow
radix_sort_keys.descending (bool, optional) – Compile-time order selector. The default is ascending key order.
temp_storage (TempStorage, optional) – Explicit block scratch. Omit for automatic allocation. Synchronize before reuse if its descriptor specifies
auto_sync=False.blocked_to_striped (bool, optional) – Compile-time selector for striped output, applied to both keys and values. The default is blocked output. Store both results with the corresponding layout.
- Returns:
Sorted keys and associated values, each retaining its input element type. Scalar operands return CuTe scalars; payload operands return fresh
ThreadDatawith the original per-thread extent. Associations are preserved, and equal selected digits retain blocked input order.- Return type:
See also
Examples
Stably order key/index pairs by their low four bits in descending order. Match striped output registers with striped stores.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def sort_digits( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() keys = coop.load(block, source, items_per_thread=items_per_thread) indices = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): indices[item] = cutlass.Int32( block.rank() * items_per_thread + item ) ordered = coop.radix_sort_keys(block, keys) pair_keys, pair_indices = coop.radix_sort_pairs( block, keys, indices, begin_bit=0, end_bit=4, descending=True, blocked_to_striped=True, ) coop.store(block, ascending, ordered) coop.store(block, descending, pair_keys, algorithm="striped") coop.store(block, positions, pair_indices, algorithm="striped") @cute.jit def launch( source: cute.Pointer, ascending: cute.Pointer, descending: cute.Pointer, positions: cute.Pointer, items_per_thread: cutlass.Constexpr, ): sort_digits( source, ascending, descending, positions, items_per_thread ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.radix_rank_keys(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_RankKeyT] | CutlassTensorSample | CutlassTensorSSASample | _RankKeyT,
- /,
- *,
- begin_bit: int = 0,
- end_bit: int | None = None,
- radix_bits: int | None = None,
- descending: bool = False,
- exclusive_digit_prefix: ThreadDataLike[int32] | ThreadDataLike[CompilerIntegerLike] | None = None,
Return stable digit ranks and optional exclusive bin prefixes.
Accepts scalar keys and readable or CuTe register payloads. An optional writable payload receives bin prefixes. Keys remain unchanged.
- Parameters:
group (ThreadGroup) – Complete physical block. Every thread participates with identical compile-time controls and payload extents. Multidimensional blocks are supported; the block tile contains at most 65,535 items.
keys (scalar, ThreadData, or CuTe register payload) –
Int32,Uint32,Int64, orUint64keys in blocked order. A scalar contributes one key per thread. CuTe register-memory tensors,TensorSSAvalues, and read-only payloads are accepted. Signed keys invert their sign bit before digit extraction.begin_bit (int, optional) – Compile-time half-open interval within the key width, selecting one through eight bits. Begin defaults to zero. Omitted end is begin plus
radix_bits, or begin plus four if both are omitted.end_bit (int, optional) – Compile-time half-open interval within the key width, selecting one through eight bits. Begin defaults to zero. Omitted end is begin plus
radix_bits, or begin plus four if both are omitted.radix_bits (int, optional) – Compile-time digit width. If
end_bitis explicit, this must equalend_bit - begin_bit.descending (bool, optional) – Compile-time selector for descending digit order. The default is ascending order.
exclusive_digit_prefix (ThreadData, optional) – Writable
Int32side output, distinct fromkeys. An omitted dtype is inferred asInt32. Each thread providesP = max(1, ceil(2**radix_bits / block_threads))items. Threadtowns ascending bin indicest * P + iin either ordering direction. A bin’s prefix counts keys with smaller digits for ascending order or greater digits for descending order. Slots past the last bin are undefined and must not be read or stored. Register tensors cannot be passed directly as this output.
- Returns:
Zero-based ranks with fixed
Int32element type. Scalar input returns a CuTe scalar; payload input returns freshThreadDatawith the input extent. Equal digits retain blocked input order. Elementiat threadtreceives the sorted position of the original key at blocked indext * items_per_thread + i.- Return type:
Int32 or ThreadData
Notes
Scratch allocation and synchronization before reuse are automatic.
See also
Examples
Rank the low four bits and retrieve the starting offset of each of the sixteen digit bins. Only the sixteen defined prefix slots are stored.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def rank_digits( source: cute.Pointer, ranks_output: cute.Pointer, prefixes_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() keys = coop.load(block, source, items_per_thread=items_per_thread) prefixes = coop.ThreadData(items_per_thread=1) ranks = coop.radix_rank_keys( block, keys, radix_bits=4, exclusive_digit_prefix=prefixes ) coop.store(block, ranks_output, ranks) coop.store(block, prefixes_output, prefixes, valid_items=16) @cute.jit def launch( source: cute.Pointer, ranks_output: cute.Pointer, prefixes_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): rank_digits( source, ranks_output, prefixes_output, items_per_thread ).launch(grid=1, block=64)
See also
Top-k selection#
- cuda.coop.cutlass.topk_min_keys(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
- valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
- temp_storage: TempStorageLike | None = None,
Select the block’s smallest keys without changing the input.
Accepts readable ThreadData and CuTe register payloads. All block threads participate, including those whose input items are outside the valid prefix.
- Parameters:
group (ThreadGroup) – Complete one-dimensional block from
this_block().keys (ThreadData or CuTe register payload) – Per-thread keys in blocked order with a positive, compile-time extent. Read-only payloads, CuTe register-memory tensors, and
TensorSSAvalues are accepted. Supported element types are signed and unsigned 8-, 16-, 32-, and 64-bit integers,Float32, andFloat64. Register inputs are converted throughThreadData.from_payload.k (integer) – Block-uniform number of requested items in
[0, N], whereN = block_threads * items_per_thread. May be static or runtime.valid_items (integer, optional) – Block-uniform length of the valid blocked input prefix, also in
[0, N]. Omit it to use the entire tile. Ifk > valid_items, all valid items are selected.temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Requested alignment is a minimum. With
auto_sync=False, synchronize the block before reusing the storage.
- Returns:
Fresh selected keys with the input element type and per-thread extent, including for CuTe register inputs. Only the first
min(k, valid_items)blocked output positions are defined; omitvalid_itemsto useNin this expression. Threadtowns positionst * items_per_thread + i. Remaining positions must not be read or stored. The selected keys are unsorted.- Return type:
Notes
Selection and ordering among equal keys are unspecified. Positive and negative floating-point zero compare equally, and selected keys retain their original bits. NaNs have no guaranteed numeric ordering.
Runtime
kandvalid_itemsaccept signed integers up to 64 bits or unsigned integers up to 32 bits. Invalid static counts fail compilation; invalid runtime counts trap before conversion to CUB’s count type. Zerokorvalid_itemsleaves no defined output positions.Examples
Select the seven smallest and seven largest keys. All threads participate, and each store writes only the seven defined positions. Neither selection sorts its output.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def select_extremes( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() keys = coop.load(block, source, items_per_thread=items_per_thread) low = coop.topk_min_keys(block, keys, k=7) high = coop.topk_max_keys(block, keys, k=7) # Selection is unsorted; only the first seven positions are defined. coop.store(block, smallest, low, valid_items=7) coop.store(block, largest, high, valid_items=7) @cute.jit def launch( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, items_per_thread: cutlass.Constexpr, ): select_extremes(source, smallest, largest, items_per_thread).launch( grid=1, block=64 )
See also
- cuda.coop.cutlass.topk_max_keys(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
- valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
- temp_storage: TempStorageLike | None = None,
Select the block’s largest keys without changing the input.
Accepts the same inputs and controls as
cuda.coop.cutlass.topk_min_keys(), selecting the largest keys. All threads in the block must participate.- Parameters:
group (ThreadGroup) – Complete one-dimensional block from
this_block().keys (ThreadData or CuTe register payload) – Fixed-size per-thread keys in blocked order. Read-only payloads, register-memory tensors, and
TensorSSAvalues are accepted. The element types are those supported bytopk_min_keys.k (integer) – Block-uniform count in
[0, block_threads * items_per_thread]. Static and runtime counts are accepted.valid_items (integer, optional) – Block-uniform length of the valid input prefix. Defaults to the whole tile and has the same range as
k.temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Synchronize before reuse if its descriptor specifies
auto_sync=False.
- Returns:
Fresh selected keys with the input element type and extent. Only the first
min(k, valid_items)blocked positions are defined, ormin(k, N)whenvalid_itemsis omitted, whereN = block_threads * items_per_thread. The selected keys are unsorted; remaining positions must not be consumed. Ties, floating-point ordering, and count validation followtopk_min_keys.- Return type:
Examples
Select the seven largest and seven smallest keys, storing only defined output positions. The selected keys are unsorted.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def select_extremes( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() keys = coop.load(block, source, items_per_thread=items_per_thread) low = coop.topk_min_keys(block, keys, k=7) high = coop.topk_max_keys(block, keys, k=7) # Selection is unsorted; only the first seven positions are defined. coop.store(block, smallest, low, valid_items=7) coop.store(block, largest, high, valid_items=7) @cute.jit def launch( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, items_per_thread: cutlass.Constexpr, ): select_extremes(source, smallest, largest, items_per_thread).launch( grid=1, block=64 )
See also
- cuda.coop.cutlass.topk_min_pairs(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
- values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
- valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
- temp_storage: TempStorageLike | None = None,
Select the smallest keys and their associated values.
Accepts readable ThreadData and CuTe register payloads for either operand. Both inputs remain unchanged. Participation, count validation, and key ordering follow
cuda.coop.cutlass.topk_min_keys().- Parameters:
group (ThreadGroup) – Complete one-dimensional block. All block threads must participate.
keys (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers,
Float32, andFloat64are supported.values (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Either operand may be read-only or use a CuTe register representation. Key and value types are independent: signed and unsigned 8-, 16-, 32-, and 64-bit integers,
Float32, andFloat64are supported.k (integer) – Block-uniform count in
[0, block_threads * items_per_thread]. Static and runtime counts are accepted.valid_items (integer, optional) – Block-uniform length of the valid blocked input prefix. Defaults to the entire tile and has the same range as
k. Ifkexceeds this count, all valid key/value pairs are selected.temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Synchronize before reuse if its descriptor specifies
auto_sync=False.
- Returns:
Fresh selected keys and associated values, each retaining its input element type and extent. Only the first
min(k, valid_items)blocked positions are defined, ormin(k, N)whenvalid_itemsis omitted, whereN = block_threads * items_per_thread. Each selected value stays with its original key. Results are unsorted, with no stability guarantee for ties; undefined tail positions must not be read or stored.- Return type:
Examples
Select seven minimum and maximum key/index pairs from a partial input tile. Original indices remain associated with their selected keys.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def select_pairs( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, smallest_indices: cute.Pointer, largest_indices: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() valid_items = 64 * items_per_thread - 3 keys = coop.load(block, source, items_per_thread=items_per_thread) indices = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): indices[item] = cutlass.Int32( block.rank() * items_per_thread + item ) low, low_indices = coop.topk_min_pairs( block, keys, indices, k=7, valid_items=valid_items ) high, high_indices = coop.topk_max_pairs( block, keys, indices, k=7, valid_items=valid_items ) coop.store(block, smallest, low, valid_items=7) coop.store(block, largest, high, valid_items=7) coop.store(block, smallest_indices, low_indices, valid_items=7) coop.store(block, largest_indices, high_indices, valid_items=7) @cute.jit def launch( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, smallest_indices: cute.Pointer, largest_indices: cute.Pointer, items_per_thread: cutlass.Constexpr, ): select_pairs( source, smallest, largest, smallest_indices, largest_indices, items_per_thread, ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.topk_max_pairs(
- group: ThreadGroup,
- keys: _ReadableThreadDataLike[_KeyT] | CutlassTensorSample | CutlassTensorSSASample,
- values: _ReadableThreadDataLike[_ValueT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- k: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any],
- valid_items: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] | None = None,
- temp_storage: TempStorageLike | None = None,
Select the largest keys and their associated values.
Accepts the same inputs and controls as
cuda.coop.cutlass.topk_min_pairs(), selecting the largest keys. Both inputs remain unchanged.- Parameters:
group (ThreadGroup) – Complete one-dimensional block. All block threads must participate.
keys (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Read-only payloads and CuTe register representations are accepted. Each operand independently uses a numeric element type supported by
topk_min_pairs.values (ThreadData or CuTe register payload) – Fixed-size payloads in blocked order, with equal per-thread extents. Read-only payloads and CuTe register representations are accepted. Each operand independently uses a numeric element type supported by
topk_min_pairs.k (integer) – Block-uniform count in
[0, block_threads * items_per_thread]. Static and runtime counts are accepted.valid_items (integer, optional) – Block-uniform length of the valid input prefix. Defaults to the entire tile and has the same range as
k. Ifkexceeds this count, all valid key/value pairs are selected.temp_storage (TempStorage, optional) – Explicit block scratch. Omit it for automatic allocation. Synchronize before reuse if its descriptor specifies
auto_sync=False.
- Returns:
Fresh selected keys and associated values, retaining each input’s element type and extent. Only the first
min(k, valid_items)blocked positions are defined, ormin(k, N)whenvalid_itemsis omitted, whereN = block_threads * items_per_thread. Associations are preserved. Results are unsorted, with no stability guarantee for ties. Floating-point ordering, count validation, and undefined tails followcuda.coop.cutlass.topk_min_keys().- Return type:
Examples
Select seven maximum and minimum key/index pairs from a partial input tile. Stores exclude the undefined output tail.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def select_pairs( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, smallest_indices: cute.Pointer, largest_indices: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() valid_items = 64 * items_per_thread - 3 keys = coop.load(block, source, items_per_thread=items_per_thread) indices = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): indices[item] = cutlass.Int32( block.rank() * items_per_thread + item ) low, low_indices = coop.topk_min_pairs( block, keys, indices, k=7, valid_items=valid_items ) high, high_indices = coop.topk_max_pairs( block, keys, indices, k=7, valid_items=valid_items ) coop.store(block, smallest, low, valid_items=7) coop.store(block, largest, high, valid_items=7) coop.store(block, smallest_indices, low_indices, valid_items=7) coop.store(block, largest_indices, high_indices, valid_items=7) @cute.jit def launch( source: cute.Pointer, smallest: cute.Pointer, largest: cute.Pointer, smallest_indices: cute.Pointer, largest_indices: cute.Pointer, items_per_thread: cutlass.Constexpr, ): select_pairs( source, smallest, largest, smallest_indices, largest_indices, items_per_thread, ).launch(grid=1, block=64)
See also
Neighbor comparisons#
- cuda.coop.cutlass.adjacent_difference(
- group: ThreadGroup,
- values: _ReadableThreadDataLike[_T] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- direction: Literal['left', 'right'] = 'left',
- valid_items: int | integer | CompilerIntegerLike | None = None,
- tile_predecessor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- tile_successor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- temp_storage: TempStorageLike | None = None,
Return blocked neighbor differences, preserving the input payload.
- Parameters:
group (ThreadGroup) – A complete
this_block()group. Every member participates in the same call; multidimensional blocks use linear thread-rank order.values (ThreadData or CuTe register payload) – Readable fixed-size payload in blocked order. Signed and unsigned 8-, 16-, 32-, and 64-bit integers, float32, and float64 are supported. Initialize every input slot, including any invalid suffix. CuTe register tensors and TensorSSA values are adapted through
ThreadData.from_payload.direction ({"left", "right"}, optional) – Compile-time choice, default
"left". Subtract the previous item from the current item for left differences, or the next item from the current item for right differences.valid_items (integer, optional) – Block-uniform count in
[0, block_size * items_per_thread]. Omit it to process the full tile. The suffix beyond this count is copied unchanged. For right differences, this argument cannot be combined withtile_successor_item.tile_predecessor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Left differences accept only a predecessor; right differences accept only a successor. Without that neighbor, the boundary input is copied unchanged.
tile_successor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Left differences accept only a predecessor; right differences accept only a successor. Without that neighbor, the boundary input is copied unchanged.
temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Omit it for automatic storage. With
auto_sync=False, synchronize the block before reusing the descriptor in another primitive.
- Returns:
Fresh values with the input dtype, extent, and minimum alignment. Invalid suffix items and a boundary without an external neighbor retain their input values. Only built-in subtraction is supported.
- Return type:
Notes
tile_predecessor_itemandtile_successor_itemgiven as NumPy or runtime scalars must match the input dtype. Plain Python integers must fit the input dtype; out-of-range values raise ValueError. Python floats require a floating-point input dtype. Finite values must fit its range; infinities and NaNs are accepted.valid_itemsmay be a Python integer, a runtime signed integer up to 64 bits, or a runtime unsigned integer up to 32 bits. An out-of-range static count raises ValueError when the kernel compiles. An out-of-range runtime count stops the kernel with a trap before conversion to CUB’s int count.See the Adjacent Difference visualization for tile boundaries.
Examples
Compute left differences using zero as the tile predecessor, then mark run heads and tails. Scratch is synchronized automatically before reuse by the second collective.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def compare_neighbors( source: cute.Pointer, differences_output: cute.Pointer, heads_output: cute.Pointer, tails_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) scratch = coop.TempStorage(auto_sync=True) differences = coop.adjacent_difference( block, values, tile_predecessor_item=0, temp_storage=scratch ) heads, tails = coop.discontinuity( block, values, mode="heads_and_tails", temp_storage=scratch ) coop.store(block, differences_output, differences) coop.store(block, heads_output, heads) coop.store(block, tails_output, tails) @cute.jit def launch( source: cute.Pointer, differences_output: cute.Pointer, heads_output: cute.Pointer, tails_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): compare_neighbors( source, differences_output, heads_output, tails_output, items_per_thread, ).launch(grid=1, block=64)
See also
- cuda.coop.cutlass.discontinuity(
- group: ThreadGroup,
- values: _ReadableThreadDataLike[_T] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- mode: Literal['heads', 'tails', 'heads_and_tails'] = 'heads',
- tile_predecessor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- tile_successor_item: int | float | int8 | uint8 | int16 | uint16 | int32 | uint32 | int64 | uint64 | float32 | float64 | CompilerScalarLike | None = None,
- temp_storage: TempStorageLike | None = None,
Flag unequal neighbors in a full blocked tile.
- Parameters:
group (ThreadGroup) – A complete
this_block()group. Every member participates in the same call; multidimensional blocks use linear thread-rank order.values (ThreadData or CuTe register payload) – Readable fixed-size payload in blocked order. Signed and unsigned 8-, 16-, 32-, and 64-bit integers, float32, and float64 are supported. CuTe register tensors and TensorSSA values are adapted through
ThreadData.from_payload.mode ({"heads", "tails", "heads_and_tails"}, optional) – Compile-time result selection, default
"heads". Heads compare the previous item with the current item; tails compare the current item with the next item. Unequal neighbors produce one, otherwise zero.tile_predecessor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Heads accept a predecessor, tails accept a successor, and
"heads_and_tails"accepts both. Without a predecessor the first head is one; without a successor the last tail is one.tile_successor_item (scalar, optional) – Block-uniform neighbor outside the tile. A runtime or NumPy scalar must match the input dtype. Python literal conversion follows the rules below. Heads accept a predecessor, tails accept a successor, and
"heads_and_tails"accepts both. Without a predecessor the first head is one; without a successor the last tail is one.temp_storage (TempStorage, optional) – Explicit block scratch descriptor. Omit it for automatic storage. With
auto_sync=False, synchronize the block before reusing the descriptor in another primitive.
- Returns:
Fresh int32 heads or tails, or
(heads, tails). Each result has the input extent and minimum alignment. The input remains unchanged.- Return type:
Notes
Partial tiles are unsupported. Padding participates in comparisons and can change the last valid tail. Only built-in inequality is supported.
tile_predecessor_itemandtile_successor_itemgiven as NumPy or runtime scalars must match the input dtype. Plain Python integers must fit the input dtype; out-of-range values raise ValueError. Python floats require a floating-point input dtype. Finite values must fit its range; infinities and NaNs are accepted.See the Discontinuity visualization for head and tail flags.
Examples
Mark both ends of each equal-value run and compute adjacent differences over the same tile.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def compare_neighbors( source: cute.Pointer, differences_output: cute.Pointer, heads_output: cute.Pointer, tails_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.load(block, source, items_per_thread=items_per_thread) scratch = coop.TempStorage(auto_sync=True) differences = coop.adjacent_difference( block, values, tile_predecessor_item=0, temp_storage=scratch ) heads, tails = coop.discontinuity( block, values, mode="heads_and_tails", temp_storage=scratch ) coop.store(block, differences_output, differences) coop.store(block, heads_output, heads) coop.store(block, tails_output, tails) @cute.jit def launch( source: cute.Pointer, differences_output: cute.Pointer, heads_output: cute.Pointer, tails_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): compare_neighbors( source, differences_output, heads_output, tails_output, items_per_thread, ).launch(grid=1, block=64)
See also
Histogram#
- cuda.coop.cutlass.histogram(
- group: ThreadGroup,
- samples: _ReadableThreadDataLike[int | uint8 | int32 | uint32 | int64 | uint64 | CompilerIntegerLike] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- bins: int,
- bins_per_thread: int = 1,
- counter_dtype: type[int | _Counter] | dtype | None = None,
- algorithm: Literal['atomic', 'sort'] = 'atomic',
- temp_storage: TempStorageLike | None = None,
Count integer samples into fresh striped bin counters.
- Parameters:
group (ThreadGroup) – A complete one-dimensional block. Every thread participates.
samples (ThreadData or CuTe register payload) – Readable fixed-size bin indices in
[0, bins). Supports uint8, int32, uint32, int64, and uint64. CuTe register tensors and immutable vectors are copied throughThreadData.from_payload.bins (int) – Positive compile-time number of bins.
bins_per_thread (int, optional) – Positive compile-time result extent.
block_size * bins_per_threadmust cover every bin. Defaults to one.counter_dtype (object, optional) – Independent counter dtype: int32 (default), uint32, int64, or uint64. Accepts NumPy and CuTe selectors. Python
intmeans int32.algorithm ({"atomic", "sort"}, optional) – Compile-time algorithm. Both preserve the input samples.
temp_storage (TempStorage, optional) – Optional scratch for intermediate counters and CUB storage. Both algorithms allocate counters in shared memory, including
"atomic". Omit the descriptor for automatic allocation. Requested alignment is a minimum. Withauto_sync=False, synchronize the block before reusing the descriptor.
- Returns:
Fresh counters in striped order: member
treceives bint + i * block_sizein sloti. Slots beyondbinsare zero. Use striped Store to write contiguous bin order.- Return type:
Notes
Each call starts from zero, even when scratch is reused. Accumulate returned counters explicitly to count several tiles. There is no
valid_itemscontrol; padding contributes samples. Sample indices must lie in[0, bins); the operation does not check their bounds. See the Histogram visualization for sample and counter layouts.Examples
Count samples with atomic and sort algorithms. Both return seventy counters followed by zero padding in striped register order. Each output allocation holds 128 counters for this 64-thread block.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def count_samples( source: cute.Pointer, atomic_output: cute.Pointer, sort_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() samples = coop.load(block, source, items_per_thread=items_per_thread) atomic_counts = coop.histogram( block, samples, bins=70, bins_per_thread=2 ) sort_counts = coop.histogram( block, samples, bins=70, bins_per_thread=2, algorithm="sort" ) coop.store(block, atomic_output, atomic_counts, algorithm="striped") coop.store(block, sort_output, sort_counts, algorithm="striped") @cute.jit def launch( source: cute.Pointer, atomic_output: cute.Pointer, sort_output: cute.Pointer, items_per_thread: cutlass.Constexpr, ): count_samples( source, atomic_output, sort_output, items_per_thread ).launch(grid=1, block=64)
See also
Run Length Decode#
- cuda.coop.cutlass.run_length_decode(
- group: ThreadGroup,
- run_values: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
- run_lengths: _ReadableThreadDataLike[_LengthT] | CutlassTensorSample | CutlassTensorSSASample,
- /,
- *,
- decoded_items_per_thread: int,
- decoded_window_offset: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] = 0,
- temp_storage: TempStorageLike | None = None,
Decode a blocked window without changing either input payload.
CuTe register-memory tensors and
TensorSSAvalues are adapted throughcuda.coop.cutlass.ThreadData.from_payload(). The window may start inside a run and may extend past the decoded stream.- Parameters:
group (ThreadGroup) – Complete one-dimensional block. Every thread must participate.
run_values (ThreadData or CuTe register payload) – Equal fixed-size per-thread payloads in blocked order. Values use the signed or unsigned 8-, 16-, 32-, or 64-bit integers,
Float32, orFloat64. Lengths use signed or unsigned integers up to 64 bits. Positive lengths precede all trailing zero padding; an all-zero tile represents an empty stream.run_lengths (ThreadData or CuTe register payload) – Equal fixed-size per-thread payloads in blocked order. Values use the signed or unsigned 8-, 16-, 32-, or 64-bit integers,
Float32, orFloat64. Lengths use signed or unsigned integers up to 64 bits. Positive lengths precede all trailing zero padding; an all-zero tile represents an empty stream.decoded_items_per_thread (int) – Positive compile-time output extent, independent of the run extent.
decoded_window_offset (integer, optional) – Block-uniform nonnegative start in the decoded stream, default zero. Member
tstarts at this offset plust * decoded_items_per_thread. Static and runtime integers up to uint64 retain their width through validation. A window starting beyond the stream contains zeros.temp_storage (TempStorage, optional) – Explicit block scratch, or omit for automatic allocation. Requested alignment is a minimum. Synchronize before reuse with
auto_sync=False.
- Returns:
Fresh payload with the run-value dtype and requested output extent. Slots beyond the decoded stream contain zero. Neither input is modified. The decoded total must fit uint32; negative lengths, misplaced zero padding, and overflow trap before decoding.
- Return type:
Notes
Both run and window tile extents must fit signed 32-bit integers. Invalid static controls fail compilation; negative runtime offsets trap before writing outputs. Each call prepares its own CUB run table. Use
cuda.coop.cutlass.run_length_decode_into()to write a full stream while preparing the table once.See the Run Length Decode visualization for windows and zero-filled tails.
Examples
Decode a 64-item window starting inside a run, then write the full decoded stream at destination offset five. A window extending beyond the stream ends with zeros.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def decode_runs( window_pointer: cute.Pointer, stream_pointer: cute.Pointer, totals_pointer: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.ThreadData(items_per_thread) lengths = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): values[item] = cutlass.Int32( block.rank() * items_per_thread + item + 10 ) lengths[item] = cutlass.Uint32(2) scratch = coop.TempStorage(auto_sync=True) window = coop.run_length_decode( block, values, lengths, decoded_items_per_thread=2, decoded_window_offset=3, temp_storage=scratch, ) coop.store(block, window_pointer, window) destination = cute.make_tensor( stream_pointer, cute.make_layout(64 * items_per_thread + 5) ) total = coop.run_length_decode_into( block, values, lengths, destination, decoded_items_per_thread=1, destination_offset=5, temp_storage=scratch, ) totals = cute.make_tensor(totals_pointer, cute.make_layout(32)) totals[block.rank()] = total @cute.jit def launch( window_pointer: cute.Pointer, stream_pointer: cute.Pointer, totals_pointer: cute.Pointer, items_per_thread: cutlass.Constexpr, ): decode_runs( window_pointer, stream_pointer, totals_pointer, items_per_thread ).launch(grid=1, block=32)
See also
- cuda.coop.cutlass.run_length_decode_into(
- group: ThreadGroup,
- run_values: _ReadableThreadDataLike[_ItemT] | CutlassTensorSample | CutlassTensorSSASample,
- run_lengths: _ReadableThreadDataLike[_LengthT] | CutlassTensorSample | CutlassTensorSSASample,
- destination: CutlassTensorSample,
- /,
- *,
- decoded_items_per_thread: int,
- destination_offset: int | signedinteger[Any] | CompilerIntegerLike | unsignedinteger[Any] = 0,
- temp_storage: TempStorageLike | None = None,
Decode a full stream into a CuTe global-memory tensor.
The prepared CUB table remains live through all internal decode windows. Both run inputs remain unchanged. The destination must not overlap either input, and its remaining capacity must hold the full decoded stream.
- Parameters:
group (ThreadGroup) – Complete one-dimensional block. Every thread must participate.
run_values (ThreadData or CuTe register payload) – Matching blocked run payloads, as in
cuda.coop.cutlass.run_length_decode().run_lengths (ThreadData or CuTe register payload) – Matching blocked run payloads, as in
cuda.coop.cutlass.run_length_decode().destination (cute.Tensor) – Contiguous one-dimensional global-memory tensor with the run-value dtype and a known static or runtime extent. All block members supply the same tensor. Bare pointers lack the capacity needed for validation.
decoded_items_per_thread (int) – Positive compile-time number of items per thread in each decode window. Each window holds
block_threads * decoded_items_per_threaditems; the final window writes only the remaining valid items.destination_offset (integer, optional) – Block-uniform nonnegative output index, default zero. Static and runtime integers up to uint64 are checked before pointer arithmetic.
temp_storage (TempStorage, optional) – Scratch covering the complete operation. Synchronize before reuse when its descriptor sets
auto_sync=False.
- Returns:
Full decoded size, available to every member. Empty input writes nothing. Insufficient capacity traps before any output write; elements outside the decoded interval remain unchanged. The total must fit uint32.
- Return type:
cutlass.Uint32
Notes
The full stream is decoded;
destination_offsetchooses where it is written. Offset and capacity validation complete before any output write. Negative lengths, misplaced zero padding, and total-size overflow followcuda.coop.cutlass.run_length_decode().See the Run Length Decode visualization for bulk output and run ordering.
Examples
Write the full decoded stream at destination offset five, alongside a register window of the same runs. Every thread receives the total decoded length.
The launcher accepts device pointers and a compile-time
items_per_threadvalue.import cutlass from cutlass import cute import cuda.coop.cutlass as coop @cute.kernel def decode_runs( window_pointer: cute.Pointer, stream_pointer: cute.Pointer, totals_pointer: cute.Pointer, items_per_thread: cutlass.Constexpr, ): block = coop.this_block() values = coop.ThreadData(items_per_thread) lengths = coop.ThreadData(items_per_thread) for item in cutlass.range_constexpr(items_per_thread): values[item] = cutlass.Int32( block.rank() * items_per_thread + item + 10 ) lengths[item] = cutlass.Uint32(2) scratch = coop.TempStorage(auto_sync=True) window = coop.run_length_decode( block, values, lengths, decoded_items_per_thread=2, decoded_window_offset=3, temp_storage=scratch, ) coop.store(block, window_pointer, window) destination = cute.make_tensor( stream_pointer, cute.make_layout(64 * items_per_thread + 5) ) total = coop.run_length_decode_into( block, values, lengths, destination, decoded_items_per_thread=1, destination_offset=5, temp_storage=scratch, ) totals = cute.make_tensor(totals_pointer, cute.make_layout(32)) totals[block.rank()] = total @cute.jit def launch( window_pointer: cute.Pointer, stream_pointer: cute.Pointer, totals_pointer: cute.Pointer, items_per_thread: cutlass.Constexpr, ): decode_runs( window_pointer, stream_pointer, totals_pointer, items_per_thread ).launch(grid=1, block=32)
See also