Debugging and Profiling#

Start debugging with small, deterministic inputs. Read results back to the host, compare against a CPU reference, then inspect generated Tile IR or profile the GPU when correctness is established.

Printing and Assertions#

cuda_tile_print! prints from inside a GPU kernel:

#[cutile::entry()]
fn debug_kernel<const S: [i32; 2]>(
    z: &mut Tensor<f32, S>,
    x: &Tensor<f32, { [-1, -1] }>,
) {
    let pid0 = program_id(0);
    let pid1 = program_id(1);
    let tile = x.load_like(z);

    cuda_tile_print!("Program ({}, {}): loaded tile\n", pid0, pid1);
    z.store(tile);
}

GPU printing is slow and serializes tile block execution. Use it for small grids and remove it before measuring performance.

cuda_tile_assert! checks conditions inside a kernel:

let n: i32 = x.shape()[0];
cuda_tile_assert!(n > 0, "expected a non-empty input");

Host Readback#

Host readback is a DeviceOp; execute it before reading the host vector:

let z_host: Vec<f32> = z
    .unpartition()
    .to_host_vec()
    .sync_on(&stream)?;

assert!(!z_host.iter().any(|x| x.is_nan()));
assert!(!z_host.iter().any(|x| x.is_infinite()));

If a fused kernel is wrong, split it into stages and read back each intermediate. Each stage should match a simple CPU implementation on a small input.

Correctness Tests#

Use minimal inputs first:

#[test]
fn small_add_matches_cpu() {
    let a = vec![1.0, 2.0, 3.0, 4.0];
    let b = vec![10.0, 20.0, 30.0, 40.0];
    let expected = vec![11.0, 22.0, 33.0, 44.0];

    let result = run_add_kernel(&a, &b);
    assert_eq!(result, expected);
}

Then compare larger random inputs against a CPU reference with an appropriate tolerance:

for (cpu, gpu) in cpu_result.iter().zip(gpu_result.iter()) {
    assert!((cpu - gpu).abs() < 1e-5, "CPU={cpu}, GPU={gpu}");
}

For numerically sensitive kernels, test edge cases: zeros, large positive values, large negative values, non-divisible shapes if supported, and known overflow-prone inputs.

Inspecting Tile IR#

print_ir = true prints the generated entry point wrapper and the Tile IR text during JIT compilation:

#[cutile::entry(print_ir = true)]
fn debug_ir_kernel<const S: [i32; 2]>(...) { ... }

dump_mlir_dir writes the compiled Tile IR text to files:

#[cutile::entry(dump_mlir_dir = "/tmp/cutile-ir")]
fn debug_ir_kernel<const S: [i32; 2]>(...) { ... }

Module-level dumps are also available with environment variables. They are written to stderr once per compiled module:

Variable	Description	Default
`CUTILE_DUMP`	Comma-separated stages to dump: `ir` (the Tile IR module text) and `bytecode` (alias `bc`; the encoded bytecode decoded back to text), or `all`. The `ast`, `resolved`, `typed`, and `instantiated` names are accepted but no code path emits them today	unset
`CUTILE_DUMP_FILTER`	Comma-separated `module::function` paths; the dumps are per module, so only the `module` part is matched. Bare function names do not exclude any module	unset

Errors and Crashes#

Most cuTile Rust errors are caught before a kernel runs:

Error	Cause	Fix
Shape mismatch	Incompatible tile shapes	Align shapes or use `reshape` / `broadcast`
Element type mismatch	Different element types in one operation	Add explicit `convert_tile()`
Invalid reduction axis	Axis outside the tile rank	Use an axis in `0..rank`
Unsupported MMA shape or dtype	No lowering for that combination	Use a supported shape and element type
Missing entry	Function is not marked with `#[cutile::entry()]`	Add the entry attribute

Runtime errors usually come from out-of-bounds accesses, toolkit issues, or invalid raw-pointer usage:

Error	Cause	Fix
CUDA error: no kernel image	Wrong GPU architecture or stale cubin	Clear cache, rebuild, verify target SM
Failed to load kernel	CUDA toolkit or driver issue	Check `nvidia-smi` and toolkit version
Out of memory	Tensor allocation or JIT memory pressure	Reduce allocation size or specialization count
Shape mismatch at runtime	Tensor size incompatible with partition	Ensure expected divisibility or bounds handling

CPU segfaults usually mean the failure happened in host-side FFI, JIT compilation, or raw-pointer lifetime management rather than inside ordinary safe tile code. Get a backtrace first:

RUST_BACKTRACE=1 cargo run
RUST_BACKTRACE=full cargo run

gdb --args ./target/debug/my_program
(gdb) run
(gdb) bt

Check the CUDA driver, CUDA Toolkit path, raw pointer lifetimes, spawned task lifetimes, and host memory use during first-launch compilation.

Debug Builds and Sanitizers#

Device debug information follows Cargo’s profile debug setting by default. With the standard profiles, cargo build enables device-debug mode and cargo build --release disables it. This is independent of debug-assertions.

For example, turn device debugging off in a development build through the workspace’s Cargo.toml:

[profile.dev]
debug = false

Cargo provides only DEBUG=true|false to build scripts, not the exact debug level. Any enabled level currently selects device-debug mode, including debug = "line-tables-only" and "limited". Device-debug mode compiles at optimization level 0, even in a release profile with debug = true. Automatic line-only mapping is not implemented. Use the explicit line override below for optimized profiling. See Cargo’s profile settings and build-script environment.

CUDA_RUST_DEBUG overrides the Cargo-derived default when building the app. No per-launch .compile_options() call is needed:

# Optimized device code with source lines for profiling.
CUDA_RUST_DEBUG=line cargo build --release

# Unoptimized device code with source locations and inline frames.
CUDA_RUST_DEBUG=full cargo build
cuda-gdb --args ./target/debug/my_program

The accepted values are none, line, and full. Unset follows Cargo; other values fail the build. Cargo tracks changes, so no cargo clean is needed. The setting is captured when the target cutile-compiler library is built, not when the proc macro is built. A package-specific profile override for cutile-compiler affects this shared default; an override only for a kernel’s crate does not. Setting the environment variable only when running an already-built binary does not change device compilation.

CompileOptions::new() and CompileOptions::default() inherit this build default. Use debug_info() to replace it for one launch. It replaces the inherited mode, so it can also turn off a build-time full default:

use cutile::tile_kernel::{CompileOptions, DebugInfoLevel};

my_kernel(args)
    .compile_options(CompileOptions::new().debug_info(DebugInfoLevel::None))
    .sync()?;

Each option is part of the JIT cache key, so the modes do not share a cached compiled kernel. Sanitizer instrumentation is a separate option:

use cutile::tile_kernel::{CompileOptions, DebugInfoLevel};

// cuda-gdb: debug information, no optimization.
my_kernel(args)
    .compile_options(CompileOptions::new().debug_info(DebugInfoLevel::Full))
    .sync()?;

// Profiler correlation: line-number information only, full optimization.
my_kernel(args)
    .compile_options(CompileOptions::new().debug_info(DebugInfoLevel::Line))
    .sync()?;

// Compute Sanitizer: memory-access instrumentation.
my_kernel(args).compile_options(CompileOptions::new().sanitize_memcheck(true)).sync()?;

debug_info(DebugInfoLevel::Full) selects device-debug mode, which passes --opt-level 0 --device-debug. The frontend also stops hoisting bounds checks out of loops. Checks it proved unnecessary or moved to launch time are unaffected; they never reach device code in any mode.
debug_info(DebugInfoLevel::Line) passes --lineinfo: source-line correlation for Nsight Compute and Nsight Systems without changing code generation. This is the option for profiling optimized kernels.
debug_info(DebugInfoLevel::None) and Line keep the current optimization level, or use level 3 when leaving device-debug mode.
opt_level(n) passes --opt-level n and keeps line information. tileiras supports device debugging only at level 0, so this also leaves device-debug mode. Whichever of debug_info and opt_level is called last decides.
sanitize_memcheck(true) passes --sanitize=memcheck for compute-sanitizer --tool memcheck.

To switch an already-built application without changing its code, set these variables when running it. Each applies after the code’s options, as if the matching setter were called last. They are read once per process:

Variable	Values	Acts like
`CUTILE_JIT_DEBUG`	`none`, `line`, `full`	`debug_info(...)`
`CUTILE_JIT_OPT_LEVEL`	`0`, `1`, `2`, `3`	`opt_level(n)`
`CUTILE_JIT_SANITIZE`	`memcheck`, `none`	`sanitize_memcheck(...)`

CUTILE_JIT_DEBUG=full cuda-gdb --args ./target/release/my_program
CUTILE_JIT_SANITIZE=memcheck compute-sanitizer --tool memcheck ./target/release/my_program

An unset variable leaves the code’s choice in place. An invalid value, or CUTILE_JIT_DEBUG=full together with CUTILE_JIT_OPT_LEVEL, fails JIT compilation with an error that names the variable.

Full debug mode describes source locations and inline frames, not Rust variables or values. Tile IR currently supports control-flow debugging of unoptimized code but not inspection of user variables; see the Tile IR debug-info documentation.

User helpers and methods retain their definition’s source file and line when inlined, including across modules. Core operations are attributed to the user’s call site. Multiple generated instructions can still map to one Rust line; this is not a one-to-one mapping, especially with optimization.

Known toolkit limitation: tileiras 13.4.59 (in the CUDA 13.4.1 CI image) and 13.4.92 can replace Rust helper locations with generated Tile IR locations and omit inline frames, even when the frontend bytecode contains the correct source information. This affects source correlation and helper backtraces. The same bytecode retains those locations with tileiras 13.3.36. CI reports this exact backend limitation while continuing to check frontend origins and surviving kernel locations; it does not waive checks for other releases, including 13.5. See #1385.

Generated const-generic array bindings use the containing function or method’s signature as their source origin. Generated reduction bodies use the reduction call’s location, including its inline caller chain.

Checks moved to the host launcher report the original access and its inline callers when validation fails, including in builds without device debug info. If several accesses share one launch check, its diagnostic retains each distinct source origin while the predicate is evaluated only once.

Profiling#

Use NVIDIA Nsight Compute (ncu, reports opened with ncu-ui) for individual kernels: throughput, occupancy, register use, and stalls. Nsight Systems (nsys, nsys-ui) shows the application timeline: CPU/GPU scheduling, transfers, synchronization, and launch gaps.

For source correlation, build with CUDA_RUST_DEBUG=line. Keep the Rust sources available on the machine where you inspect the report. Profile the built executable, not the Cargo build itself:

CUDA_RUST_DEBUG=line cargo build --release
ncu --target-processes all ./my_cutile_program
ncu --set full -o profile_report ./my_cutile_program
ncu-ui profile_report.ncu-rep

Replace ./my_cutile_program with your binary’s path, such as ./target/release/my_program. An ERR_NVGPUCTRPERM report means this machine’s GPU performance-counter access needs to be enabled by its administrator; changing line-info settings will not fix that.

Use Nsight Systems for CPU/GPU scheduling:

nsys profile ./my_cutile_program
nsys-ui report.nsys-rep

Look for launch gaps, unnecessary synchronization, memory transfer overlap, and whether independent kernels actually overlap on separate streams.

Tile IR display is separate from Rust source-line correlation. NVIDIA lists the Tile IR Source view as a new feature in Nsight Compute 2026.3. If it is missing, record the Nsight Compute and CUDA Toolkit versions, compile options, cubin, and report before diagnosing a metadata problem. The standalone Tile IR dumps above remain useful independently of that view.

Regression Checks#

The repository tests source provenance and compile-option cache separation without a GPU:

cargo test -p cutile --test debug_info
cargo test -p cutile --test jit_env_overrides
bash scripts/test_debug_defaults.sh

The second command requires jq. It checks Cargo profiles, independent debug assertions and build-dependency settings, explicit overrides, and the line-only mapping limitation. It also checks that changing the runtime environment does not change the built default. Additional checks are opt-in:

# Requires tileiras, nvdisasm, and readelf, but no GPU.
cargo test -p cutile --test debug_info cubin_has_rust_lines_and_inline_frames -- --ignored

# Requires a supported GPU and tileiras.
cargo test -p cutile --test debug_info kernel_executes_in_all_debug_modes -- --ignored

# Also requires cuda-gdb and timeout; stops at a helper's source line and
# checks its nested inline backtrace before completing the kernel.
cargo test -p cutile --test debug_info cuda_gdb_stops_in_cross_file_helper -- --ignored

Debugging Checklist#

Shapes match the operation and launch partition.
Tensor sizes are compatible with the partition shape.
Element types match or are explicitly converted.
Small inputs match a CPU reference.
Numerically sensitive code handles overflow and underflow.
Raw pointers outlive all GPU work that uses them.
print_ir shows the expected Tile IR operations.
Profiles are captured after correctness checks pass.

Review Performance for optimization strategies or Interoperability for custom CUDA kernels.

Migrating compile options to v0.5.0#

Use debug_info(DebugInfoLevel::Full) instead of device_debug(true), and debug_info(DebugInfoLevel::Line) instead of lineinfo(true). Use debug_info(DebugInfoLevel::None) to disable both kinds of debug information. The old public opt_level, device_debug, and lineinfo fields are replaced by codegen: CodegenMode; prefer the builder setters when constructing options.

Setter order is significant: .debug_info(Full).opt_level(3) selects optimized code, while .opt_level(3).debug_info(Full) selects full debugging at level zero. Runtime CUTILE_JIT_* overrides are applied after these programmatic options.