Achieving High Performance#

Below is some general advice to help achieve high performance using NPPDx.

General advice#

  • Begin with the defaults: look at suggested_tile_size and the derived block_dim before hand-tuning.

  • Measure the effect of tile and block variations: compute-bound and memory-bound algorithms may require different tile and block sizes and shapes.

  • Test with representative image sizes: small images often under-fill the GPU regardless of tile and block configurations.

Memory management#

  • Prefer Register-only API: when every processing step is pointwise, the Register-only API is faster. Any step with a non-zero halo requires the fused kernel to use the Shared-memory API.

  • Reuse compatible TileStorage buffers: for sequential shared-memory processing steps, ping-pong between TileStorage input and output buffers instead of allocating additional buffers for every step, and reserve separate temp storage for operations that require it, such as resize.

  • Aligned memory allocation: ingest and exgest are fastest when naturally aligned. This means each tile fits within the image and the packed layout is aligned to memory:

    • base pointers and row strides aligned to the packing format’s natural row-block size (the byte interval at which that format’s channel layout repeats).

    • image width and height are even.

    Tiles that cross the image edge or are misaligned fall back to slower per-pixel handling. That cost is usually negligible for large images, but can be significant for small images where those edge tiles are a large fraction of the total. Large-halo cases can also cause slower performance.

Kernel fusion#

  • Fuse steps: keep intermediate data in registers or shared memory instead of writing and re-reading global memory between processing steps.

  • Fusion reduces kernel launches: global-memory traffic mostly happens during ingest and exgest.

  • Watch resource usage: Adding fused steps can increase register pressure, shared-memory use, and code size. This is fine until registers start to spill or there isn’t enough shared memory for all of the steps.

  • Split when fusion costs too much: Multiple kernels may outperform a single fused kernel when:

    • the fused kernel has a very large cumulative halo

    • some steps require global image state or cross-tile coordination

    • there is fusion of several high register-usage processing steps

    Tip

    If splitting is necessary, use the optimized rgb16 storage format between launches to preserve data during the global memory round trip. Note: An affine channel map step can be used to pre-scale data to prevent significant range clipping or precision loss.

  • Start from the examples: Find the example closest to the target use case and examine both a more-fused and a less-fused case before deciding on a final strategy.

Tuning knobs: TileSize, BlockDim#

Because the fused kernel is fully composed at compile time, experiment with the user-facing knobs per architecture and workload instead of writing one generic kernel.

Knob

Guidance

TileSize <W, H>

Larger tiles:

  • Raise arithmetic intensity and compute efficiency.

  • Raise shared-memory footprint and register pressure.

  • Better amortize work spent on halos.

  • May not fit for a specific SM architecture.

Smaller tiles:

  • More likely to fit and leave occupancy headroom on a given SM.

  • Will have less compute performance.

BlockDim <...>

Balance occupancy against elements_per_thread register usage.

Advanced#

  • use NVIDIA Nsight Compute: Check occupancy or cudaOccupancyMaxActiveBlocksPerMultiprocessor when deciding whether a register-limited or halo-heavy kernel should be split.

Further reading#