Developer Guide#

CuTe compiles your kernel and its control flow. For each cooperative primitive, cuda.coop.cutlass generates a C++ device function that calls CUB. NVRTC compiles those functions to LTO-IR, which CuTe links into the kernel before it runs. The implementation calls these generated functions providers.

During tracing, CuTe runs the kernel’s Python code with symbolic values and builds its intermediate representation (IR). A cooperative call at this stage adds device instructions to that representation; the GPU runs them later. Lowering adapts the call and its values to the generated provider’s argument and result types. A provider request records which C++ function is needed, and a session collects those requests for one trace.

The overview introduces groups, per-thread items, and the public import. The Programming Guide covers writing kernels and choosing operation controls. Here, an executable prefix-sum example shows how a call becomes a compiled device function and where to look when changing the implementation.

For a hands-on tour, follow the Debugger Walkthrough. It uses a 128-thread tile copy to show import, planning, code generation, linking, and repeated execution.

If CUTLASS and its CuTe DSL are new to you, read Introduction to CUTLASS for cuda.coop Developers first. It explains how the DSL traces a kernel, how that differs from Dynamo, Numba, and Triton, and which DSL facilities this package relies on. For the CuTe integration mechanisms alone, without the package’s abstractions, see Minimal CuTe DSL Integration Example. It calls cub::BlockReduce from a CuTe kernel in one file and lists the questions open for CUTLASS reviewers.

Following a prefix sum#

The executable Scan example launches one (8, 4, 2) block. By default, each of its 64 threads owns two items, for a tile of 128 integers. The host entry point checks the following kernel against NumPy:

    @cute.kernel
    def scan_tiles(
        source: cute.Pointer,
        destination: cute.Pointer,
        items_per_thread: cutlass.Constexpr,
    ):
        group = coop.this_block()
        storage = coop.TempStorage(
            sharing="shared", alignment=64, auto_sync=True
        )
        payload = coop.load(
            group,
            source,
            items_per_thread=items_per_thread,
            algorithm="transpose",
            temp_storage=storage,
        )
        scanned = coop.exclusive_scan(
            group, payload, initial_value=7, temp_storage=storage
        )
        # Scan returns a new payload. The input retains its original values.
        coop.store(
            group,
            destination,
            scanned,
            algorithm="transpose",
            temp_storage=storage,
        )

    @cute.jit
    def launch(
        source: cute.Pointer,
        destination: cute.Pointer,
        items_per_thread: cutlass.Constexpr,
    ):
        scan_tiles(source, destination, items_per_thread).launch(
            grid=1, block=_BLOCK
        )

Load fills payload with consecutive input values. Scan returns a new payload whose item at position i is seven plus the sum of all inputs before i in blocked order. Input [0, 1, 2, 3, ...] therefore produces [7, 7, 8, 10, ...]. Store writes those prefixes to the destination. Every thread calls all three primitives. They reuse one TempStorage descriptor, with auto_sync=True synchronizing scratch reuse.

The complete example includes allocation, launch, cleanup, and an independent NumPy result check. From the repository root, run it in the compatible environment described in the programming guide:

python python/cuda_coop_cutlass/examples/scan.py

CuTe owns the @cute.kernel and @cute.jit functions, their Python control flow, and the launch. Each primitive is a device call inside the same kernel.

Compiler requirements#

The CuTe compiler must support external NVIDIA LTO-IR linking and trace-finalization hooks that run after tracing and before module hashing. It must provide access to the active compiler environment, exact block, grid, and cluster dimensions, and launch flags. _compiler/_runtime.py checks for the required Python interfaces during compiler initialization. Compilation and execution tests check that the interfaces work together with NVRTC and the final linker.

Compiler phases and handoffs#

Read each figure from top to bottom. These steps run on the host. CuTe traces the Python kernel with symbolic values. It compiles and links the result before the GPU executes it.

The phase map shows where cooperative calls enter this process. Select Python classes to see the classes, data records, and module functions that implement each step. Function labels include parentheses; a data record carries information between steps.

CUTLASS compilation phases, from tracing cooperative calls through provider collection, bundle compilation, scratch materialization, and final linking.

Tracing records calls. Finalization supplies their definitions and completes deferred scratch allocations.#

CUTLASS compilation phases arranged vertically, from tracing through provider compilation, scratch materialization, linking, and execution.

Tracing records calls. Finalization supplies their definitions and completes deferred scratch allocations.#

CUTLASS phases with concrete implementation names. BundleSession collects provider requests. The trace finalizer obtains a BundleCompilation and materializes DeferredTempStoragePlan records before CuTe links the kernel.

Classes and records describe ownership and data. Module functions perform much of the compilation work.#

Detailed CUTLASS phases in a vertical layout, including BundleSession, provider request records, bundle compilation, and deferred scratch plans.

Classes and records describe ownership and data. Module functions perform much of the compilation work.#

Each cooperative call is planned as CuTe traces it. The lowering already has the operand types and exact launch facts needed to select its provider. It adds a request to BundleSession, then emits a symbolic call. CuTe’s end-of-trace hook runs after these requests have been collected and before device linking.

The next figure expands this boundary. It shows a nonempty session on a provider-cache miss, with deferred scratch. NVRTC compiles the provider bundle; CuTe compiles the enclosing kernel and arranges the final link. The figures show the device-LTO path that produces a cubin. CuTe supplies kernel LTO-IR and provider LTO-IR to nvJitLink.

Actor swimlanes show CuTe tracing, cuda.coop collecting providers, NVRTC returning LTO-IR and scratch layouts, and CuTe arranging the final device link before launch.

The bundle and layout queries share one NVRTC program. The final device link is a later step.#

CUTLASS compilation handoffs in time order, with CuTe, cuda.coop, NVRTC, and the device linker identified at each step.

The bundle and layout queries share one NVRTC program. The final device link is a later step.#

Implementation-level handoffs through BundleSession, the trace-finalization hook, BundleCompilation, deferred storage planning, and GPU module link-libraries.

The finalizer supplies an artifact path and completes scratch operands. It does not launch the kernel.#

Detailed CUTLASS handoffs in a vertical layout, from provider requests and symbolic calls through NVRTC results, scratch operand replacement, and final linking.

The finalizer supplies an artifact path and completes scratch operands. It does not launch the kernel.#

Equivalent provider requests share one generated wrapper. Scratch events retain each traced use, because separate uses can need separate slices. A valid cache entry supplies both the LTO-IR and the queried layouts, so that path does not call NVRTC. A session with no providers needs no bundle. Separate trace modules keep separate sessions. A trace module can contain several kernels; deferred scratch is allocated separately for each kernel.

Calls before compiled provider code exists#

A provider request determines a symbol name and an argument/result contract before NVRTC runs. The renderer uses that name for the C++ definition. The lowering uses the same name and contract to emit cute.ffi calls. This step needs a symbol, not a loaded device address.

At finalization, _trace_finalize_hook() attaches the compiled bundle to the GPU module’s link-libraries attribute. CuTe computes its module hash after the hook returns, builds the module, then checks its compile cache. The device linker resolves the call against its definition and can optimize it with the kernel. The provider can be inlined, so it need not remain a separate function in the final cubin. Loading that cubin makes the kernel available to launch.

Scratch addresses follow a separate path. During tracing, calls that need deferred scratch use placeholder address and size operands. NVRTC supplies the exact C++ size and alignment. The finalizer inserts shared-memory allocations and replaces those operands before device compilation finishes. It does not replace a placeholder provider function pointer.

What to inspect at each boundary#

Trace data and finalization data#

Object or record

What it tells you

GroupLoweringPlan

Selected implementation, participation, result, storage, and synchronization requirements for one cooperative call.

Provider request, such as _CubScanRequest

Plan plus the CuTe types and ABI choices needed to render a wrapper and emit its matching call.

BundleSession

Unique provider requests and ordered scratch events for one trace module. A snapshot restores this bookkeeping after failed emission; it does not undo emitted IR.

BundleCompilation

Path to provider LTO-IR and exact C++ layouts indexed by requirement. A cache hit and a fresh compilation return the same kind of result.

DeferredTempStoragePlan

Allocation size, alignment, and per-use bindings. Materialization inserts allocations and replaces each use’s address and size.

Compiler integration#

Importing cuda.coop.cutlass checks the CuTe runtime through _compiler/_runtime.py. A cooperative call reads the active CuTe compiler environment and emits into the enclosing kernel trace. Import itself does not compile a kernel.

The private _compiler modules own trace sessions, provider bundles, launch facts, finalization, cache artifacts, and deferred storage. The _planning modules select CUB specializations and their participation, result, storage, and synchronization contracts. The _headers package resolves the configured source tree or installed header bundle and toolkit.

Primitive families#

A family groups related primitives and their implementation. Scan includes scan, inclusive_scan, exclusive_scan, inclusive_sum, and exclusive_sum. Its public entry points and type declarations live in _group_scan.py and _group_scan.pyi; planning lives in _planning/group/scan.py and CuTe adaptation in _lowering/_scan.py. Paths in this guide are relative to python/cuda_coop_cutlass/cuda/coop/cutlass unless stated otherwise.

Planning the CUB call#

For Load, _group_load_store.py first validates the group, payload, and controls. It classifies a control as omitted, a compile-time constant, or a runtime value. The lowering then reads the CuTe memory operand’s element type, or takes the dtype the caller declared for it, and checks that its layout exposes the contiguous pointer required by CUB.

The planner uses those arguments and the kernel’s launch dimensions to choose a CUB specialization. It also determines which threads must participate and what scratch storage and barriers the call needs. The lowering puts the plan and CuTe scalar type into a provider request, which the renderer uses to generate the wrapper.

In the example, algorithm="transpose" selects CUB Block Load and Store for a 64-thread block with two integers per thread. Thread rank t owns tile items 2 * t and 2 * t + 1. Block Scan computes exclusive sums in that blocked order. The generated wrappers and scratch plans reflect each primitive’s algorithm, element type, and items per thread.

The renderer generates a device wrapper with an ordinary callable symbol. CuTe’s cute.ffi emits the call to that symbol. Load uses a result buffer to return its items, then updates the ThreadData values after the FFI call is successfully emitted. Store passes the existing items to its provider. The plan identifies equivalent requests so they can share one generated function and cached artifact.

Other primitive families use the same approach. For Reduce, the planner selects CUB BlockReduce or WarpReduce, checks scalar or per-thread payload support, and records scratch requirements. Block reductions accept an explicit TempStorage descriptor; warp reductions use compiler-owned storage. The Reduce lowering generates the wrapper and adapts CuTe’s values to its arguments. The returned scalar is defined only at group rank zero.

Exact launch facts#

_compiler/_launch.py reads the compiler-provided exact block, grid, and cluster dimensions and the cooperative/cluster launch flags. The adapter records their origin in the LaunchFacts object. If a dimension or flag is unavailable, it stays unknown in that object. Planning fails if the primitive needs the missing value.

For the prefix sum, block dimensions determine both the CUB specialization and linear rank: x + block_x * (y + block_y * z). Warp primitives also need the exact block size to establish complete physical warps and allocate one scratch slice per group. Cluster primitives need consistent cluster dimensions and launch mode.

Do not substitute maxntid for exact dimensions: an upper bound does not prove the number of participating threads. The adapter does not infer launches from Python frames or a user-maintained launch description.

One provider bundle per trace#

Each trace has a provider session keyed by its compile options and MLIR module. The session deduplicates equivalent requests and records deferred scratch uses. Load, Scan, and Store in the example contribute to the same bundle; finalization compiles the three providers together.

cuda.coop.cutlass registers one persistent trace-finalization hook with CuTe. The hook runs for every trace on that DSL. It finds the session belonging to the module being finalized and leaves unrelated or nested sessions alone. The hook renders the requests into one C++ translation unit, compiles it to NVIDIA LTO-IR, and attaches the artifact to each gpu.module in the trace through its link-libraries attribute. The CuTe compiler and final linker then resolve the cute.ffi calls while compiling the enclosing kernel.

Header and toolkit discovery use the helpers under _headers. The compiler libraries for the selected toolkit are loaded before importing the NVRTC bindings, and the loaded NVRTC version is checked against that selection. A mismatch reports an error before provider compilation. The final linker must also accept the generated LTO-IR; successful NVRTC compilation alone does not establish that compatibility.

The cache identity includes the rendered source, header identity, compiler options, architecture, and resolved toolkit/compiler information. Cache loads validate the artifact and any recorded scratch layouts. Cache writes use a lock and atomic replacement. The LTO-IR file remains available until the final linker has read it. Finalization removes stale managed library paths from persistent compiler options so the next trace cannot link an earlier bundle by mistake.

Failed emission restores the queued requests and pending scratch uses saved in the provider-session snapshot. It does not undo emitted IR or payload changes. Finalization removes only its own session. Lifecycle tests exercise failed compilation or linking followed by a successful retry, as well as repeated and nested compilation. These checks matter because one Python process can compile many kernels through the same CuTe DSL.

Scratch allocation after tracing#

Direct, striped, and vectorized Load/Store do not use scratch. Their provider ABI omits a storage pointer, and passing a TempStorage descriptor does not add a reuse barrier or change the storage-free provider’s identity.

Calls with deferred scratch record its use while tracing. The finalizer adds sizeof and alignof probes to the same NVRTC program as the provider bundle. The resulting C++ layouts determine the allocation; the adapter does not estimate storage sizes from the element count. Other providers, including Warp Scan, declare their shared storage inside the generated C++ wrapper and do not need this placeholder-replacement step.

Deferred events let one explicit descriptor serve several calls. Shared storage uses enough capacity for the largest requirement, while exclusive storage assigns separate slices to distinct call sites. Requested alignment is a minimum. An explicit capacity that cannot accommodate its uses is an error. Once the layouts are known, the finalizer materializes shared-memory allocations and replaces the trace’s storage placeholders.

TempStorage.reserve records an event with a known element size and alignment, and returns an ordinary CuTe tensor built from an address placeholder. It participates in the same descriptor layout as primitive scratch. Finalization resolves these views even when the kernel has no cooperative provider calls. A descriptor with reservations requires auto_sync=False for every use; finalization also checks earlier and later primitive events before materializing its allocation.

TempStorage.release(dtype=None) consumes the descriptor and returns its complete region as a one-dimensional native CuTe tensor. The default dtype is cutlass.Uint8. Inferred size resolves during finalization; exclusive slices, their padding, and explicit extra capacity are included. The region must be nonempty, its size divisible by the element size, and its guaranteed alignment sufficient for the requested type. The descriptor must be constructed inside the kernel. For a typed release without preceding primitive or reservation uses, specify alignment of at least the dtype’s element size on the constructor.

Release adds no allocation or synchronization. Complete previous accesses and asynchronous operations before using overlapping bytes. All aliases of the descriptor become unusable on the executing path, including calls to reserve, sync, release, and storage-free primitives. The finalizer checks reachable branch and loop paths, so a conditional release prevents a later use after the branches join. Mutually exclusive releases are valid, as is a fresh descriptor constructed and consumed within each iteration. Runtime selection between different descriptor instances is rejected when release is involved; it cannot silently select the Python trace’s template. Identity tokens and use markers support this analysis without device-side lifetime flags.

Exclusive slices still need reuse synchronization when one call site runs again in a loop. Explicit descriptors default to auto_sync=False for both sharing modes, so the kernel must call storage.sync() before reuse. Set auto_sync=True to insert automatic trailing synchronization. Compiler-managed scratch synchronizes automatically. Physical and logical warp primitives use independent per-group storage and the appropriate warp mask rather than a block barrier. See the programming guide for each family’s explicit-storage support and participation rules.

Finding the implementation#

Paths below are relative to python/cuda_coop_cutlass/cuda/coop/cutlass. Planning describes primitive semantics; lowering adapts CuTe values and emits provider calls.

Source map#

Area

Starting points

Header and toolkit discovery

_headers/__init__.py, _headers/_identity.py, _headers/_toolkit.py

Compiler validation and launch facts

_compiler/_runtime.py, _compiler/_launch.py

User-facing values and groups

_thread_data.py, _thread_group.py, _temp_storage.py, and their .pyi files

Family planning and lowering

_planning/group/ and _lowering/; public entry points live in the corresponding _group_*.py modules

Provider sessions and finalization

_compiler/_state.py, _compiler/_finalize.py

Rendering, compilation, and artifacts

_compiler/_rendering.py, _compiler/_bundle.py, _compiler/_nvrtc.py, _compiler/_cache.py

Scratch layout, materialization, and lifetime

_compiler/_layout.py, _compiler/_storage.py, _compiler/_lifetime.py

Debugger Walkthrough#

Follow a tile copy through a Python debugger to see the planner choose a CUB specialization, CuTe emit the device calls, and NVRTC compile their providers. These stops are in the host Python code doing the compilation. The GPU executes the resulting kernel later; a Python breakpoint does not stop a GPU thread.

An example to debug#

Open docs/python/coop_cutlass/debugger_walkthrough.py in your checkout, or download the example. It needs no command-line arguments. Its kernel and launcher are:

    @cute.kernel
    def copy_tile(
        source: cute.Pointer,
        destination: cute.Pointer,
        items_per_thread: cutlass.Constexpr,
    ):
        block = coop.this_block()
        scratch = coop.TempStorage(auto_sync=True)
        items = coop.load(
            block,
            source,
            items_per_thread=items_per_thread,
            algorithm=algorithm,
            temp_storage=scratch,
        )
        coop.store(
            block, destination, items, algorithm=algorithm, temp_storage=scratch
        )

    @cute.jit
    def launch(
        source: cute.Pointer,
        destination: cute.Pointer,
        items_per_thread: cutlass.Constexpr,
    ):
        copy_tile(source, destination, items_per_thread).launch(
            grid=1, block=128
        )

The default algorithm is "direct" and items_per_thread is 2. Each of the 128 threads copies two integers, so the default tile contains 256 items. The kernel and launcher specialize the item-count argument with cutlass.Constexpr; pass --items-per-thread to change it. scratch is a descriptor: Direct Load and Store ignore it and allocate no shared memory. The same descriptor shows scratch reuse in the transpose pass.

The complete example allocates device buffers with the CUDA Driver API, wraps their addresses in CuTe pointers, and frees them before returning. It compiles the launcher once, then calls the resulting function twice:

            compiled = cute.compile(
                launch, source_pointer, destination_pointer, items_per_thread
            )
            for iteration in range(2):
                destination.fill(-1)
                _check(
                    driver.cuMemcpyHtoD(
                        destination_device,
                        destination.ctypes.data,
                        destination.nbytes,
                    )
                )
                compiled(source_pointer, destination_pointer)
                _check(driver.cuCtxSynchronize())
                _check(
                    driver.cuMemcpyDtoH(
                        destination.ctypes.data,
                        destination_device,
                        destination.nbytes,
                    )
                )
                np.testing.assert_array_equal(destination, source)
                print(f"Launch {iteration + 1} ({algorithm}): copy verified")

Each iteration resets the device output to -1, launches the copy, synchronizes, and checks every value against NumPy. Here, cute.compile makes compilation an explicit step before either launch. The two calls to compiled reuse that kernel.

A direct call to a CuTe @cute.jit function can trace again to compute the module’s cache key, even when compiled code is reusable. Retaining the compiled callable makes the two GPU launches independent of that tracing path.

Configure VS Code#

Open this CCCL checkout as the VS Code folder. Use Python: Select Interpreter to select an environment satisfying the CUTLASS compiler requirements and the programming guide’s installation requirements. A CUTLASS package that imports successfully may still lack an interface needed by this walkthrough. Install the Python and Python Debugger extensions where the program runs, including the remote side when using Remote SSH.

Add this configuration to .vscode/launch.json. If the file already exists, add the entry to its configurations list:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "cuda.coop CUTLASS: Debug active Python file",
      "type": "debugpy",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "cwd": "${workspaceFolder}",
      "justMyCode": false,
      "env": {
        "PYTHONPATH": "${workspaceFolder}/python/cuda_coop_cutlass",
        "CUDA_COOP_CUTLASS_CCCL_ROOT": "${workspaceFolder}",
        "CUDA_COOP_CUTLASS_SOURCE_DUMP_DIR": "${workspaceFolder}/build/coop-cutlass-debug-sources"
      }
    }
  ]
}

justMyCode: false permits stepping into library code. PYTHONPATH and CUDA_COOP_CUTLASS_CCCL_ROOT select this checkout’s Python sources and C++ headers. cute.compile explicitly requests CuTe compilation. When run as a script, the example also selects a fresh temporary provider-cache directory and removes it on exit. A new process with that empty provider cache lets each debug session reach NVRTC.

On a machine with several GPUs, add CUDA_VISIBLE_DEVICES to env. Use the toolkit configured for your compatible compiler environment.

Set the first breakpoint on import cuda.coop.cutlass as coop in the example. Keep that file active, select the named configuration in Run and Debug, and press F5. ${file} selects the active editor file when the session starts. F10 steps over, F11 steps into, Shift+F11 steps out, and F5 continues. Debug Console expressions use the selected Call Stack frame. See the VS Code Python debugging documentation for those controls.

The stops below identify executable statements by function name, so they remain useful as line numbers change. Set each breakpoint before continuing past the stage that calls it. Avoid calling primitives from Debug Console: they can emit additional IR into the paused trace.

Initial import and compiler validation#

The first stop is just before import cuda.coop.cutlass as coop; the example has already imported CuTe. In _compiler/_runtime.py, set a breakpoint in validate_cutlass_runtime before continuing through the package import. Inspect the compiler environment and supported runtime hooks. This validation establishes the available interfaces; compilation and device execution remain separate steps.

Disable the import breakpoints and stop on compiled = cute.compile(...) in the example. Evaluate:

coop.__file__
cutlass.__file__
source[:8]
destination[:8]
algorithm

coop.__file__ should point inside your checkout. cutlass.__file__ identifies the dependency selected by the interpreter. The arrays start with 0, 1, ..., 7 and eight -1 values respectively, and algorithm is "direct". Compilation and both launches are still ahead of you.

Follow the launch dimensions#

Continue from cute.compile until CuTe traces the kernel. Cooperative calls execute directly in this trace and read its exact launch dimensions. Paths for the following stops are relative to python/cuda_coop_cutlass/cuda/coop/cutlass/.

In _group_load_store.py:_resolve_group(), break on resolved = _resolve_primitive_group_from_launch(...). At that stop, inspect:

operation
launch.exact_block_dim
launch.exact_grid_dim
launch.provenance

The first operation is "load", with block (128, 1, 1) and grid (1, 1, 1). The provenance names the compiler’s launch-facts API. Step into current_kernel_launch_facts() on a fresh pass if you want to follow that dependency boundary. These dimensions come from .launch(grid=1, block=128) and determine the CUB specialization.

From a primitive call to a plan#

Set a breakpoint in _lowering/_load_store.py:_make_request() on return _CubLoadStoreRequest(plan, value_type). Continue and inspect:

kind
value_type
plan.target
dict(plan.implementation.template_arguments)
plan.participation
plan.temp_storage
plan.synchronization

The plan selects CUB_BLOCK, cutlass.Int32, 128 threads, two items per thread, and ::cub::BLOCK_LOAD_DIRECT. Participation requires the full block. Storage ownership and the storage-reuse barrier are NONE, even though the example supplied scratch.

Set the next breakpoint in provider_load() in the same file, on _state.register_request(request). Inspect request.cpp_type, request.symbol_name, runtime_types, and runtime_args. The C++ type specializes cub::BlockLoad. The symbol names the device wrapper that will implement it. Both runtime-control lists are empty: this copy has no valid-prefix count, offset, or fill argument.

Step over request registration. It records the request in a session owned by this trace and arranges for a finalization hook. No C++ has been compiled yet.

Emit the device calls#

Still in provider_load(), stop on ffi(...) after scratch_types, scratch_args = _scratch_arguments(...). Both scratch lists are empty. The FFI signature has two pointers: the source address and a per-thread result buffer containing two integers.

Step over the complete FFI call and the following payload assignments. CuTe has emitted a call into its IR, and output now holds the CuTe values read from that result buffer. You are inspecting compiler values; the GPU has not loaded a particular thread’s integers yet.

Continue to the plan breakpoint for Store. It selects ::cub::BLOCK_STORE_DIRECT. In provider_store(), break on its ffi(...) call and inspect values and request.symbol_name. Store passes the destination pointer and the two item values. Unlike Load, it needs no result buffer. Disable the plan and emission breakpoints after inspecting both operations.

Finalize the trace and compile the providers#

Before continuing from Store, set a breakpoint in _compiler/_finalize.py:_trace_finalize_hook() on source = _rendering.render_bundle_source(requests). This statement comes after the checks that select this module’s provider session, so unrelated modules do not stop here. Inspect:

len(requests)
[request.symbol_name for request in requests]
session.deferred_temp_storage_event_list()
print(module)

There are two requests, one for Load and one for Store, and no deferred storage events. The IR contains calls to the symbols you saw in the lowerings. Step over the rendering assignment and inspect source. Both wrappers belong to one C++ translation unit. The Load wrapper builds a two-item C++ array, calls BlockLoad::Load, and writes the result buffer; Store builds its array from the two scalar arguments and calls BlockStore::Store.

In _compiler/_bundle.py:_compile_bundle_source(), break on options = _nvrtc.compiler_options(context, arch). Continue, inspect context.include_dirs and context.toolkit_root, then step over the assignment. options contains C++17, relocatable device code, -dlto, the target architecture, and the include directories. This is also a useful stop for detecting headers or toolkit libraries from the wrong installation.

Before continuing, set a breakpoint in _compiler/_nvrtc.py:_compile_ltoir() on error, program = nvrtc.nvrtcCreateProgram(...). At that stop, source contains both providers and prepared is None for this storage-free copy. Step through nvrtcCompileProgram and its error check, then the nvrtcGetLTOIRSize and nvrtcGetLTOIR calls. blob contains the provider LTO-IR. The full C++ is also saved under build/coop-cutlass-debug-sources for inspection outside the debugger.

Finish the launch and observe reuse#

Disable the compiler breakpoints and set one on compiled(source_pointer, destination_pointer) in the example. Continue. cute.compile has returned a callable; the first launch is about to execute it with iteration == 0. Continue again to reach the second launch with iteration == 1. The first verification message has printed, and destination has been reset to -1 on both host and device.

Re-enable the plan and NVRTC breakpoints, then continue. Calling compiled executes the existing kernel without tracing or provider compilation. Neither breakpoint should fire, and the second verification message should print. Calling cute.compile again would request another CuTe compilation, though the provider cache could supply its LTO-IR. Start a new debug session to repeat the tour; editing a file while paused does not replace the function already loaded in the current process.

A second pass: shared scratch and synchronization#

Add "args": ["--algorithm", "transpose"] to the launch configuration and start a new debug session. From a terminal, the equivalent is:

python docs/python/coop_cutlass/debugger_walkthrough.py --algorithm transpose

The kernel, launch dimensions, and expected output are unchanged. At the plan stop, ALGORITHM now selects BLOCK_LOAD_TRANSPOSE or BLOCK_STORE_TRANSPOSE. Storage ownership is CALLER, and the storage-reuse barrier is BLOCK. The FFI signatures gain three scratch arguments: a shared-memory address, its capacity in bytes, and the automatic-synchronization flag.

Set these additional breakpoints before reaching Load:

  • In _compiler/_storage.py:register_deferred_temp_storage_event(), break on session.add_deferred_temp_storage_event(...). Inspect primitive_name, requirement_key, smem_addr_placeholder, and size_placeholder. The address and size are fresh IR placeholders; NVRTC has not yet supplied the C++ scratch layout. Load and Store record separate uses of the same TempStorage descriptor.

  • In _compiler/_finalize.py:_trace_finalize_hook(), break on plans = _storage.plan_deferred_temp_storage_events(...). Inspect compilation.layouts. NVRTC has compiled both providers and recovered their exact C++ sizeof and alignof requirements from the same program. Step over planning and inspect plans: there is one shared allocation, sized for the larger requirement. Its two bindings have byte offset zero. The size and alignment come from this build; do not copy fixed numbers from another toolkit.

  • In _compiler/_storage.py:materialize_deferred_temp_storage_plans(), break on smem_ptr = allocator.allocate(...). Inspect plan.size_in_bytes, plan.alignment, and plan.bindings. Step through allocation and operand replacement to see the recorded calls acquire their actual address and capacity.

At the link-input stop, compare print(module) with the Direct version. There is now a shared allocation and the calls carry scratch operands. Open the dumped C++ and find temp_storage_auto_sync. Each wrapper ends with a conditional __syncthreads(); this example passes a true flag, so Load finishes using the workspace before Store reuses it.

CUTLASS emits the trailing barrier in the provider wrapper and patches deferred storage operands after obtaining the C++ layouts. ThreadData still holds the per-thread payload; TempStorage describes the shared workspace used during each primitive. Continue to verify both launches of the transpose copy.

If a breakpoint does not stop#

  • Check coop.__file__ at the first launch. The executing module and breakpoint must belong to the same checkout. Inspect cutlass.__file__ too if a required compiler interface is missing.

  • Use the named launch configuration with justMyCode: false. A hollow breakpoint can remain pending until its module loads. Put it on an executable statement rather than a decorator or def line.

  • If compiler initialization is skipped, check the example’s import order. Read any dependency error or compiler initialization warning before investigating later compiler stops.

  • If NVRTC is skipped, restart the complete script. Its temporary provider cache is created only under if __name__ == "__main__". Check that the provider directory is new and that you set the breakpoint before cute.compile. A stop in _compile_bundle_source() can show a provider-cache hit. A source dump alone does not prove NVRTC ran: the backend writes the dump before looking up a cached provider.

  • If a later launch retraces, check that you are calling compiled. This walkthrough retains the callable returned by cute.compile and invokes it twice; calling the decorated launch directly follows CuTe’s JIT path instead.

Working on a primitive#

Start with the C++ overload and a small kernel exercising it. Check the family plan for payload layout, participation, result ownership, and storage requirements. Keep the public entry point and its .pyi declaration aligned.

Adapt the plan in _lowering. The lowering must agree with its provider wrapper about argument types, result buffers, and scratch operands. Register the typed request with the current session and preserve rollback if emission fails. Load demonstrates a newly returned payload; Scan and Reduce show additional result contracts and controls. Reuse the existing renderer compiler initialization, bundle finalizer, and storage planner when their contracts express the operation.

Checking a change#

Start with the affected family under python/cuda_coop_cutlass/tests. Unit tests check plan selection, argument contracts, and state transitions. Compile tests exercise real CuTe traces, launch metadata, FFI emission, and lifecycle recovery. Runtime tests compare results and layouts against independent references, including partial tiles and repeated scratch reuse. For storage-policy routing, see unit/test_load_store_plans.py.

For a Load/Store or lifecycle change, the following are useful entry points from the repository root with the package and compatible compiler installed:

python -m pytest -q python/cuda_coop_cutlass/tests/unit/test_compiler.py
python -m pytest -q python/cuda_coop_cutlass/tests/compile/test_compiler_lifecycle.py
python -m pytest -q python/cuda_coop_cutlass/tests/runtime/test_block_load_store.py
python -m pytest -q python/cuda_coop_cutlass/tests/runtime/test_block_algorithms.py

The compile and runtime examples above require a compatible CUDA environment and GPU. Inspect skips when assessing coverage: an absent optional compiler does not establish a passing compiler integration. Fresh-process and installed-artifact tests check namespace coexistence, module origins, import diagnostics, and typing. Run affected planning tests when changing specialization or participation rules.

To inspect the generated C++, set CUDA_COOP_CUTLASS_SOURCE_DUMP_DIR before compiling a kernel. The dump includes the source on provider-cache hits too. CUDA_COOP_CUTLASS_PROVIDER_CACHE_DIR selects the provider artifact cache when a test needs an isolated cache directory. See Configuration for these settings and header-selection controls.

For generated-code claims, inspect the final linked cubin with cuobjdump. runtime/test_block_load_store.py checks that direct Load/Store leave no provider calls, shared accesses, or barriers. The final-cubin check in runtime/test_block_algorithms.py tests striped, vectorize, and transpose. It requires shared memory and barriers only for transpose. The final-cubin checks in runtime/test_warp_load_store.py and runtime/test_logical_warp_load_store.py reject provider calls and block barriers for physical and logical Warp operations. They forbid shared memory and warp synchronization for storage-free algorithms. Transpose must use shared memory, except that the width-one test leaves allocation unconstrained.

The final-cubin check in runtime/test_reduce.py confirms that the CUB Reduce routes leave no provider calls. It does not check shared memory, registers, or barriers.

The final-cubin check in runtime/test_scan.py rejects provider calls for block and logical-Warp Scan, and block barriers for the logical-Warp path. It saves resource reports but does not check them.

The final-cubin checks in runtime/test_exchange.py and runtime/test_shuffle.py reject provider calls for block and logical-Warp Exchange and for array and scalar Shuffle. The Exchange check also rejects block barriers on the logical-Warp path. Neither check examines shared memory. runtime/test_group_hierarchy.py checks that mapped-group queries leave no calls, barriers, warp synchronization, or shared memory.

The final-cubin check in runtime/test_merge_sort.py rejects provider calls for a partial block sort and a logical-Warp sort. It also rejects block barriers on the logical-Warp path. It does not check shared memory.

Provider source or intermediate PTX alone cannot prove the final result. Use Compute Sanitizer race checks for changes to scratch allocation or synchronization. Numerical correctness, generated-code evidence, and public-package qualification require separate checks.