Minimal CuTe DSL Integration Example#

cuda.coop.cutlass integrates with the CUTLASS CuTe DSL through a handful of mechanisms that are easy to lose among planners, thread groups, payloads, and caches. This page isolates them. One script, cutlass_integration_example.py, calls cub::BlockReduce from a CuTe kernel using only the DSL’s own interfaces, NVRTC, and the CCCL headers. It imports nothing from cuda.coop.

The intended readers are CUTLASS developers reviewing whether this is how the DSL is meant to be extended, and cuda.coop developers who want the mechanisms without the abstractions. The closing section lists the questions we would like CUTLASS reviewers to answer. Introduction to CUTLASS for cuda.coop Developers explains the DSL’s tracing model that the script builds on.

Run it on Linux from a CCCL checkout with a compatible CUTLASS 4.9 CuTe DSL, CUDA 13 driver and toolkit, cuda-bindings, and NumPy. This example was validated with the retained 4.9 candidate and requires its launch-facts and trace-finalization interfaces; see cuda.coop.cutlass for installation and qualification details.

With those prerequisites available:

python docs/python/coop_cutlass/cutlass_integration_example.py
# block sum verified: 3008

CUDA_COOP_CUTLASS_EXAMPLE_PRINT_IR=1 python docs/python/coop_cutlass/cutlass_integration_example.py

The second command prints the finalized MLIR module from inside the hook. CUDA_COOP_CUTLASS_CCCL_ROOT selects another CCCL checkout for the headers. CUDA_COOP_CUTLASS_EXAMPLE_ARTIFACT_DIR selects an existing parent directory for the example’s temporary LTO-IR files; they are removed when the process exits. The package’s persistent cache is configured separately with CUDA_COOP_CUTLASS_PROVIDER_CACHE_DIR; see Configuration.

Compilation and execution#

cute.compile drives the host-side steps below. The trace-finalize hook supplies the CUB wrapper and completes the kernel’s scratch operands before CuTe links the two LTO-IR inputs. Calling the returned callable then launches the GPU kernel.

Four columns show CuTe tracing the kernel, the example collecting the CUB call and finalizing scratch, NVRTC returning wrapper LTO-IR and scratch layout, and nvJitLink combining kernel and wrapper LTO-IR. Labeled arrows show each handoff, followed by GPU execution below the host lanes.

Columns show ownership; arrows show the data passed between components. On narrow screens, scroll horizontally to follow all four columns.#

What the script uses from the DSL#

Interface

Status

Purpose

@cute.kernel, @cute.jit, .launch, cute.compile

public

Write, compile, and launch the kernel.

cute.ffi

public

Emit a call to a symbol defined later.

cutlass.memory.SmemAllocator

public

Allocate CUB scratch in shared memory.

cute._get_launch_facts()

private

Read the exact block shape while tracing the body.

CuTeDSL._get_dsl().register_trace_finalize_hook

private

Run after tracing, before the module hash and device link.

gpu.module link-libraries attribute

internal

Hand the compiled LTO-IR to the device link.

cutlass._mlir (ir, arith, llvm)

internal

Trace placeholders, insert the allocation, replace operands.

dsl.envar.arch

internal

Select the NVRTC target from the DSL’s own setting.

The kernel#

The kernel sums one int per thread across a (32, 4, 1) block and thread zero stores the total. block_sum is the integration; everything else is ordinary CuTe.

@cute.kernel
def sum_block(source: cute.Pointer, destination: cute.Pointer):
    tidx, tidy, _ = cute.arch.thread_idx()
    rank = tidx + tidy * _BLOCK[0]
    inputs = cute.make_tensor(source, cute.make_layout(_THREADS))
    output = cute.make_tensor(destination, cute.make_layout(1))
    total = block_sum(inputs[rank])  # 1. Traced like any other CuTe call.
    if rank == 0:
        output[0] = total


@cute.jit
def launch(source: cute.Pointer, destination: cute.Pointer):
    sum_block(source, destination).launch(grid=1, block=_BLOCK)


Steps 1 to 3: the trace-time entry point#

The DSL compiles by executing the kernel body with symbolic values, so block_sum runs while CuTe traces sum_block. It reads the launch shape, registers the finalize hook, traces two placeholder constants for the scratch, and emits the call:

def block_sum(value: cutlass.Int32) -> cutlass.Int32:
    """Emit a call to a CUB block sum while CuTe traces the kernel.

    ``value`` wraps an MLIR value of the kernel body. The returned ``Int32``
    wraps the call result, which ``cub::BlockReduce::Sum`` defines only in
    thread zero.
    """

    # 2. Exact launch facts. ``.launch(grid=..., block=...)`` attaches them to
    #    the kernel operation before tracing its body, so the block shape is
    #    a Python tuple here. An upper bound such as maxntid would not do.
    facts = cute._get_launch_facts()
    if facts.exact_block_dim is None:
        raise RuntimeError("block_sum requires a static block shape")
    request = _Request(facts.exact_block_dim)
    _requests[request.symbol] = request

    # 4. Make sure the finalize hook runs for this trace. Registering the
    #    same callable twice is a no-op.
    CuTeDSL._get_dsl().register_trace_finalize_hook(_finalize_trace)

    # 5. Scratch placeholders. The C++ layout is unknown until NVRTC runs,
    #    so trace two i32 constants and remember them with their kernel.
    smem_addr = arith.constant(T.i32(), 0)
    smem_bytes = arith.constant(T.i32(), 0)
    _scratch_uses.append(
        _ScratchUse(request, _enclosing_kernel(), smem_addr, smem_bytes)
    )

    # 3. Declare a private prototype in the gpu.module and emit func.call.
    #    No definition exists yet; the device linker resolves it later.
    result = cute.ffi(
        name=request.symbol,
        params_types=[cutlass.Int32, cutlass.Uint32, cutlass.Int32],
        return_type=cutlass.Int32,
    )(value, cutlass.Uint32(smem_addr), cutlass.Int32(smem_bytes))
    return cutlass.Int32(result)


def _enclosing_kernel() -> ir.Operation:
    """Walk from the insertion point to the kernel that owns the scratch."""

    op = ir.InsertionPoint.current.block.owner
    while op is not None:
        operation = getattr(op, "operation", op)
        if operation.name == "cuda.kernel" or (
            operation.name == "lir.func"
            and "gpu.kernel" in operation.attributes
        ):
            return operation
        op = operation.parent
    raise RuntimeError("block_sum must be called inside a @cute.kernel body")


Three details matter here.

The block shape is exact. KernelLauncher.launch attaches a cutlass_launch_facts attribute to the kernel operation before it traces the body, so _get_launch_facts() returns (32, 4, 1) as Python integers. CUB’s BlockReduce is a template on those dimensions, and a mismatch between the template and the launch would be silent corruption. The DSL also derives nvvm.reqntid from the same configuration.

The call is a plain ``func.call``. cute.ffi declares a private func.func prototype in the enclosing gpu.module on first use and emits the call. The symbol is used verbatim, so the C++ definition is extern "C". DSL numerics wrap MLIR values; cutlass.Int32(result) wraps the call’s result for the caller.

The scratch operands are placeholders. Nothing in Python knows sizeof(BlockReduce::TempStorage) yet. The entry point traces two arith.constant 0 : i32 values, passes them as the address and size, and records them with the enclosing kernel operation for later replacement.

Step 4: the trace-finalize hook#

After the kernel body and the host function have returned, CuTe calls each registered hook with the DSL, the finished module, and the function name. The module hash, the compile-cache lookup, and the device link all come later, so the hook can still complete the module:

def _finalize_trace(dsl, module: ir.Module, function_name: str) -> None:
    """Compile requests and complete the module before CuTe hashes it.

    CuTe calls every registered hook once per trace, after the Python body
    returns and before the module hash, the compile-cache lookup, and the
    device link. Everything emitted during tracing is visible here and can
    still be changed.
    """

    del function_name
    requests, uses = list(_requests.values()), list(_scratch_uses)
    _requests.clear()
    _scratch_uses.clear()
    if not requests:
        return  # A trace without block_sum calls needs nothing from us.

    source = _PREAMBLE + "".join(
        _WRAPPER.format(symbol=r.symbol, cpp_type=r.cpp_type) for r in requests
    )
    # The DSL already knows the target, for example "sm_120a".
    arch = "compute_" + dsl.envar.arch.removeprefix("sm_")
    ltoir, layouts = _compile_ltoir(source, arch, [r.probe for r in requests])
    digest = hashlib.sha256(ltoir).hexdigest()[:16]
    path = os.path.join(_ARTIFACTS.name, f"{digest}.ltoir")
    Path(path).write_bytes(ltoir)

    _allocate_scratch(uses, {r.symbol: layouts[r.probe] for r in requests})
    _attach_link_library(module, path)
    if os.environ.get("CUDA_COOP_CUTLASS_EXAMPLE_PRINT_IR"):
        print(module)


def _attach_link_library(module: ir.Module, path: str) -> None:
    """Name the LTO-IR file on every gpu.module.

    CuTe merges the attribute into its ``link-libraries`` compile option and
    passes the files to the device linker with the kernel's own LTO-IR.
    """

    # CuTe generally emits one gpu.module containing all kernels in a trace.
    # Visit every GPU module, as CuTe's link-library collection does.
    for op in module.body.operations:
        if op.name != "gpu.module":
            continue
        paths = {path}
        if "link-libraries" in op.attributes:
            paths.update(attr.value for attr in op.attributes["link-libraries"])
        op.attributes["link-libraries"] = ir.ArrayAttr.get(
            [ir.StringAttr.get(item) for item in sorted(paths)]
        )


The hook renders the C++ for every distinct block shape it saw, compiles it, allocates scratch, and names the LTO-IR file on the gpu.module. CuTe merges that attribute into its link-libraries compile option and passes the file to nvJitLink together with the kernel’s own LTO-IR. The DSL documents the attribute in terms of LLVM bitcode; its compiler accepts the NVRTC LTO-IR container as a link input, and the link inlines the wrapper.

The generated C++ contains a wrapper function and a layout-probe template. The wrapper uses __cvta_shared_to_generic to convert the 32-bit shared-window address into a generic pointer while retaining the compiler’s knowledge that it points to shared memory. It constructs the CUB object on that storage and ends with the barrier that makes the scratch reusable. The probe template is explained with step 5:

_PREAMBLE = f"""\
#include <cub/block/block_reduce.cuh>

// Layout probe (step 5). NVRTC evaluates the template arguments while it
// compiles the wrappers, and nvrtcGetLoweredName returns the mangled
// instantiation, which spells sizeof and alignof as decimal integers.
template <unsigned long long Size, unsigned long long Alignment>
__device__ unsigned char {_PROBE} = 0;
"""

_WRAPPER = """\
extern "C" __device__ int {symbol}(int value, unsigned int smem_addr,
                                   int smem_bytes) {{
  using reduce_t = {cpp_type};
  using storage_t = typename reduce_t::TempStorage;
  if (smem_bytes < (int) sizeof(storage_t) ||
      (smem_addr & (alignof(storage_t) - 1)) != 0) {{
    asm volatile("trap;");
  }}
  // CuTe passes a 32-bit shared-window address; CUB wants a generic pointer.
  auto* const generic_addr = __cvta_shared_to_generic(smem_addr);
  auto& storage = *reinterpret_cast<storage_t*>(generic_addr);
  int result = reduce_t(storage).Sum(value);
  __syncthreads();  // The scratch may be reused by a later call.
  return result;
}}
"""

NVRTC compiles the translation unit as relocatable device code with -dlto. The same program answers the layout query:

def _compile_ltoir(
    source: str, arch: str, probes: list[str]
) -> tuple[bytes, dict[str, tuple[int, int]]]:
    """Compile the wrappers to LTO-IR and read back the probed layouts.

    Relocatable device code with ``-dlto`` yields an LTO-IR container, the
    input kind CuTe's device link already consumes for the kernel itself, so
    the linker can inline the wrapper into the kernel.
    """

    program = _check(
        nvrtc.nvrtcCreateProgram(source.encode(), b"wrappers.cu", 0, [], [])
    )
    try:
        for probe in probes:
            _check(nvrtc.nvrtcAddNameExpression(program, probe.encode()))
        options = [
            b"--std=c++17",
            b"--relocatable-device-code=true",
            b"-dlto",
            f"--gpu-architecture={arch}".encode(),
            *(f"--include-path={path}".encode() for path in _include_dirs()),
        ]
        if nvrtc.nvrtcCompileProgram(program, len(options), options)[0]:
            log = bytearray(_check(nvrtc.nvrtcGetProgramLogSize(program)))
            _check(nvrtc.nvrtcGetProgramLog(program, log))
            raise RuntimeError(f"NVRTC failed:\n{log.decode(errors='replace')}")
        layouts = {}
        for probe in probes:
            lowered = _check(nvrtc.nvrtcGetLoweredName(program, probe.encode()))
            # For example _Z20example_layout_probeILy32ELy4EE -> (32, 4).
            match = re.search(r"ILy(\d+)ELy(\d+)EE", str(lowered))
            layouts[probe] = (int(match.group(1)), int(match.group(2)))
        blob = bytearray(_check(nvrtc.nvrtcGetLTOIRSize(program)))
        _check(nvrtc.nvrtcGetLTOIR(program, blob))
        return bytes(blob), layouts
    finally:
        nvrtc.nvrtcDestroyProgram(program)


def _include_dirs() -> list[str]:
    """Select this checkout's CCCL headers, then the CUDA runtime headers."""

    root = Path(
        os.environ.get("CUDA_COOP_CUTLASS_CCCL_ROOT")
        or Path(__file__).resolve().parents[3]
    )
    if not (root / "cub" / "cub" / "version.cuh").is_file():
        raise RuntimeError(
            f"{root} is not a CCCL checkout; set CUDA_COOP_CUTLASS_CCCL_ROOT"
        )
    from cuda.pathfinder import find_nvidia_header_directory

    cuda_include = find_nvidia_header_directory("cudart")
    if cuda_include is None:
        raise RuntimeError("CUDA runtime headers not found; set CUDA_HOME")
    return [
        str(root / "cub"),
        str(root / "thrust"),
        str(root / "libcudacxx" / "include"),
        str(cuda_include),
    ]


Step 5: shared memory after the fact#

nvrtcAddNameExpression registers the probe instantiation &example_layout_probe<(sizeof(T::TempStorage)), (alignof(T::TempStorage))> before compilation, and nvrtcGetLoweredName returns its mangled name afterwards, for example _Z20example_layout_probeILy24ELy4EE. The two integers are the size and alignment, evaluated by the same compiler that will lay out the struct. No device code runs and no second compilation is needed.

With the layout known, the hook allocates the scratch at the kernel’s entry block and rewrites the traced placeholders:

def _allocate_scratch(
    uses: list[_ScratchUse], layouts: dict[str, tuple[int, int]]
) -> None:
    """Allocate CUB scratch at each kernel entry and patch the traced calls.

    ``SmemAllocator`` only emits ``memref.alloca`` operations. The CuTe
    compiler lays them out with the kernel's own shared memory and sizes the
    launch automatically. The traced placeholders become dead constants.
    """

    by_kernel: dict[ir.Operation, list[_ScratchUse]] = {}
    for use in uses:
        by_kernel.setdefault(use.kernel_op, []).append(use)
    for kernel_op, kernel_uses in by_kernel.items():
        entry_block = kernel_op.regions[0].blocks[0]
        with ir.InsertionPoint.at_block_begin(entry_block):
            allocator = SmemAllocator()
            for use in kernel_uses:
                size, alignment = layouts[use.request.symbol]
                pointer = allocator.allocate(size, alignment)
                address = llvm.ptrtoint(T.i32(), pointer.to_llvm_ptr())
                use.smem_addr.replace_all_uses_with(address)
                use.smem_bytes.replace_all_uses_with(
                    arith.constant(T.i32(), size)
                )


SmemAllocator keeps no state. allocate emits a cute.memref.alloca tagged with smem.partition_id; the compiler lays out all such allocations in the kernel, including the user’s own, and the host code reads the total through cute.kernel_smem_size when it builds the launch configuration. That is why the allocation can be inserted after tracing without touching the launch. llvm.ptrtoint produces the 32-bit shared-window address the wrapper expects.

The alternative, which cuda.coop.cutlass also uses for wrapper-owned storage, is to declare __shared__ storage inside the C++ wrapper and let the device link place it. The deferred route is needed when a caller-supplied TempStorage descriptor must be shared or sliced across several calls, and when its size must be visible to CuTe’s own accounting.

The finalized module#

This excerpt is the module printed by CUDA_COOP_CUTLASS_EXAMPLE_PRINT_IR=1 on the CUTLASS 4.9 candidate for sm_120a, trimmed to the lines the integration produced or depends on. Names and attributes vary between DSL versions.

gpu.module @kernels attributes {"link-libraries" = [".../92e8e7ee6bc67574.ltoir"]} {
  lir.func public @kernel_cutlass_sum_block_...(%arg0: !cute.ptr<i32, gmem, align<16>>, ...)
      attributes {cutlass_launch_facts = {exact_block_dim = array<i64: 32, 4, 1>,
                                          exact_grid_dim = array<i64: 1, 1, 1>, ...},
                  gpu.kernel, nvvm.reqntid = array<i32: 32, 4, 1>, ...} {
    // Inserted by the hook at the entry block: 24 bytes, 4-byte alignment.
    %smem = cute.memref.alloca() {smem.partition_id = 1 : i32} : !memref_smem_i8
    %iter_0 = cute.get_iter(%smem) : !memref_smem_i8
    %0 = builtin.unrealized_conversion_cast %iter_0 : !cute.ptr<i8, smem, align<4>> to !llvm.ptr<3>
    %1 = llvm.ptrtoint %0 : !llvm.ptr<3> to i32
    %c24_i32 = arith.constant 24 : i32
    ...
    %7 = cute.memref.load(%view, %coord) : (!memref_gmem_i32, !cute.coord<"?">) -> i32
    // The traced placeholders, now unused.
    %c0_i32 = arith.constant 0 : i32
    %c0_i32_8 = arith.constant 0 : i32
    // The call now carries the real address and size.
    %8 = func.call @example_block_sum_i32_32x4x1(%7, %1, %c24_i32) : (i32, i32, i32) -> i32
    ...
  }
  func.func private @example_block_sum_i32_32x4x1(i32, i32, i32) -> i32
}
func.func @cutlass_launch_Ptrgmem_Ptrgmem(...) attributes {llvm.emit_c_interface} {
  %0 = cute.kernel_smem_size @kernels::@kernel_cutlass_sum_block_... : i64
  ...
  %3 = cuda.launch_cfg.create<...> (blockDim = (%c32_i32, %c4_i32, %c1_i32),
                                    dynamicSmemBytes = %0, gridDim = (...), ...)
  %4 = cuda.launch_ex @kernels::@kernel_cutlass_sum_block_...<%3> (%arg0, %arg1) ...
}

The linked cubin#

Running the script with CUTE_DSL_KEEP=cubin and CUTE_DSL_DUMP_DIR set keeps the final cubin. On the same machine, cuobjdump --dump-sass shows that the device link inlined the wrapper: the listing contains no CALL and no example_block_sum symbol, one REDUX.SUM for the warp reductions, and two BAR.SYNC, one inside CUB and one from the wrapper’s trailing __syncthreads(). Scratch accesses use LDS/STS and global input/output accesses use LDG/STG; there are no generic LD/ST instructions. The ELF symbol table still contains _Z20example_layout_probeILy24ELy4EE as a weak one-byte device object: the probe instantiation survives the link. cuda.coop.cutlass’s runtime tests make the same no-call check with cuobjdump for each primitive family.

What the package adds beyond this#

The script keeps one global request list and one scratch allocation per call. cuda.coop.cutlass keeps the same five mechanisms and adds the bookkeeping a library needs:

  • Sessions per trace. BundleSession is keyed by the DSL’s compile options object and the MLIR module, so nested @cute.jit calls, several kernels in one module, and repeated compilations stay separate. A snapshot restores the queued requests if a lowering fails part way.

  • Planning. A planner selects the CUB specialization, participation rules, result ownership, and scratch requirements from the group, dtype, controls, and launch facts, and derives a deterministic symbol name from that plan. Equivalent requests share one wrapper.

  • Scratch policy. TempStorage descriptors let several calls share one allocation or receive exclusive slices, with explicit or automatic reuse barriers, typed reservations, and a lifetime check across branches and loops. Warp primitives size one storage instance per logical warp.

  • Array results. Load returns several values per thread through a cute.make_rmem_tensor buffer whose pointer is an extra argument; Store passes each item as a scalar argument.

  • Artifacts. The LTO-IR is cached on disk under a key that includes the rendered source, header identity, compiler options, and architecture; a cache hit skips NVRTC and still supplies the recorded layouts.

  • Toolkit binding. The NVRTC and nvJitLink libraries are preloaded from the toolkit that owns the selected headers, and the NVRTC version is checked before compilation.

Questions for CUTLASS reviewers#

  1. Is register_trace_finalize_hook together with the gpu.module link-libraries attribute the intended extension point for externally compiled device code? Is an NVRTC LTO-IR container, rather than LLVM bitcode from cute.ffi(source=BitCode(...)), a supported link input going forward?

  2. cute._get_launch_facts() is private. Is reading exact block, grid, and cluster shapes from the cutlass_launch_facts attribute the right contract, and will its schema remain versioned?

  3. The hook mutates the finished module: it inserts SmemAllocator allocations at the kernel entry block and calls replace_all_uses_with on traced placeholders. Is this acceptable, or is there a preferred way to request shared memory whose size is only known after an external compilation?

  4. SmemAllocator is a dsl_user_op constructed inside the hook rather than inside the user’s trace. Does that interact badly with the user’s own allocations, partitions, or the smem.hint_smem_base_uniform attribute?

  5. The wrappers contain __syncthreads() and are inlined by the device link. Should the func.func prototype or call carry attributes such as convergent or noinline, and does cute.ffi(inline=...) express that?

  6. The probe template leaves a weak one-byte __device__ symbol in the cubin. Is there a cleaner way to obtain sizeof and alignof from an NVRTC compilation that CuTe will consume?

  7. Are there constraints on the NVRTC version that produces the LTO-IR relative to the LLVM and nvJitLink used by the DSL? The package binds NVRTC to the toolkit that supplies the CUDA runtime headers and checks its version, but the compatibility rule itself is not documented.

Testing#

tests/runtime/test_cutlass_integration_example.py runs the script in a subprocess with an isolated interpreter, checks the printed verification line, and asserts that cuda.coop.cutlass was never imported:

python -m pytest -q python/cuda_coop_cutlass/tests/runtime/test_cutlass_integration_example.py