Benchmark Results#

Pre-computed CSVs consumed by the Sphinx docs build. The shipped numbers under this directory were produced on NVIDIA H100 (Hopper) GPUs and cover three modules (neighbor list, DFT-D3 dispersion, electrostatics) across two chemical systems (CsCl, NH₃) and three scaling modes.

The original snapshot was collected on 8 July 2026 under run ID b890cd6794884c1f9e9b143e104fa6da and source fingerprint 695f537dbf4dc6b6e33911363250f629af2629f489a38c583a27c5370373b770. Its 18 reportable CSVs contain 3,356 rows: 3,324 successful measurements and 32 explicit EL capacity-limit rows. The EL failures comprise 21 OutOfMemoryError rows, 9 strict-PME JaxRuntimeError rows, and 2 SkippedAfterOOM rows for JAX Ewald. Another 148 configured JAX NH3 EL rows are absent from that snapshot.

That collection embeds its fingerprint in software_context. The measured source tree was clean at Git head 66b2aa334a9f2d6b138bf0d1ee87da0e09055593; later documentation-data edits do not alter a timed kernel, callable, grid, or CSV value. The software context was Python 3.13.9, Torch 2.12.0+cu126, JAX/JAXlib 0.9.0.1, Warp 1.13.0, CUDA 12.6, and ALCHEMI Toolkit-Ops 0.4.0. Collection used NVIDIA H100 80 GB HBM3 GPUs (compute capability 9.0) with driver 535.216.03. A kernel, public API, timing-boundary, or grid change requires replacement of every affected case.

The current 18-file matrix contains all 3,512 planned rows: 3,499 successful measurements and 13 Ewald OOM rows. The six EL files contain 592 rows: all 296 PME measurements, 283 Ewald measurements, and the 13 OOM records. Every planned EL case was attempted; no unmeasured points remain.

The refreshed Torch PME collection uses run ID ec9f9176-c20b-4c6b-a2d3-efae9282f598; the JAX PME and Ewald collection uses 28ea42c8-c3b9-4a9c-96dc-b96a346325d0. Both use source fingerprint 4dfd278da46af31f54f8b7671e891073f8b10f8fa2cb28e7cf92bfeddd543638. PME uses spline order five, a 25% total-point FFT snap budget, float64, accuracies 1e-4 and 1e-6, and a 9 Å real-space cutoff cap. Each row measures the full energy, forces, and charge-gradient workload with five groups of ten timed calls after three warmups, using arithmetic-mean aggregation; setup and neighbor construction are excluded.

These collections used separate NVIDIA H100 NVL allocations reporting 99,996,401,664 memory bytes (93.13 GiB usable), compute capability 9.0, and unavailable driver metadata. Both used Python 3.12.3, Torch 2.14.0+cu130, Warp 1.17.0, CUDA 13.0, and ALCHEMI Toolkit-Ops 0.5.0. JAX/JAXlib was 0.10.1 for the JAX collection and 0.11.2 in the Torch collection’s software context.

Torch Ewald retains all 148 successful measurements from the earlier refresh, run ID da2e3992-31bf-4906-ae02-d3747f02a55a, source fingerprint 0796a4ffc622cdc7ab846f2e8cf88b0b52fccae4dcfe32a3140b3bb9818aab8e. It used the Torch collection’s software versions on NVIDIA H100 GPUs reporting 102,132,088,832 memory bytes (95.12 GiB usable), compute capability 9.0, and unavailable driver metadata. Comparisons therefore combine separate recorded GPU allocations and software contexts. The 12 NL/D3 files retain their original hardware, software, and all 2,920 rows unchanged by this refresh.

See the per-module doc pages for how to read the plots and how to reproduce:

../neighborlist.md
../dftd3.md
../electrostatics.md

File naming#

Names follow the scheme emitted by benchmarks.suite_utils.make_csv_name(module, system, mode):

{module}-{system}-{mode-slug}.csv

Where module ∈ {nl, d3, el}, system ∈ {cscl, nh3}, and mode-slug ∈ {system-size-scaling, constant-workload-scaling, batch-scaling}. Example: nl-cscl-system-size-scaling.csv.

The current reportable NL/D3/EL suite uses only root-level nl-*.csv, d3-*.csv, and el-*.csv files. Separate dynamics and segment-operation CSVs may also remain at the root for their own docs pages. Outputs from the optional extended electrostatics runner use electrostatics_benchmark_*.csv names and a different schema; they are not read by the reportable plot generation path.

CSV schema#

Emitted by benchmarks.suite_utils.build_result:

Column	Type	Description
`system`	str	`cscl` or `nh3`
`scaling_mode`	str	`system_size`, `constant_workload`, or `batch_scaling`
`method`	str	NL strategy (`naive_scalar`, `naive_tile`, `cell_list_atom_centric`, `cell_list_pair_centric`, `cluster_tile`, plus batch-prefixed concrete APIs where applicable), `dftd3` (D3), or `pme` / `ewald` (EL)
`backend`	str	`torch`, `jax`, or `warp` where supported
`atoms_per_system`	int	Atoms in one system
`batch_size`	int	Number of systems in the batch
`total_atoms`	int	`atoms_per_system` × `batch_size`
`time_us_per_atom`	float	Mean μs per atom across the batch timing
`throughput_atoms_per_sec`	float	Derived throughput
`mem_delta_mb`	float	Torch CUDA allocator delta from the pre-timing measurement call (MB); NaN for JAX
`mem_peak_gb`	float	Torch CUDA allocator peak (GB); NaN for JAX
`timing_runs`	int	Number of timed calls per timing group
`timing_batches`	int	Number of timing groups; five in the current EL collection
`timing_batch_aggregation`	str	Aggregation across timing groups; `arithmetic_mean` in the current EL collection
`warmup_runs`	int	Number of untimed warmup calls before measurement
`timing_method`	str	Timing path used for the row, such as `torch_cuda_events`, `jax_wall_block_until_ready`, or the serial fallback `jax_wall_block_each`
`timing_method_real`	str	Added by EL; `not_measured` unless component profiling is enabled
`timing_method_reciprocal`	str	Added by EL; `not_measured` unless component profiling is enabled
`compile_policy`	str	Compile/warmup policy; shipped rows use `warmup_excluded`
`success`	bool	`False` rows are filtered by the plotter
`error`	str	Concise failure or skip message for `success=False` rows
`error_type`	str	Stable failure class, such as `OutOfMemoryError`, `JaxRuntimeError`, `UnsupportedConfiguration`, `SkippedByPolicy`, or `SkippedAfterOOM`
`failure_stage`	str	Optional; populated by failure paths that can identify the setup or timing stage that raised the error
`cutoff`	float	Added by NL and D3
`configured_max_neighbors`	int	Added by NL; neighbor-matrix capacity selected before the measured call
`max_neighbors`	int	Added by NL; width of the allocated or returned dense neighbor matrix
`max_total_cells`	int	Added by cell-list NL rows; allocated cell-grid capacity
`cell_list_min_cells`	int	Added by NL backends that expose this cell-grid metadata; minimum per-axis cell count used by the selected strategy
`total_neighbor_pairs`	int	Added by NL; backend-specific pair/storage count (dense paths report allocated slots, while direct Warp rows report populated pairs)
`accuracy`	float	Added by EL
`time_d3_us_per_atom`	float	Added by D3 (excludes NL build time)
`neighbor_setup_method`	str	Added by D3; neighbor-list setup API used before timing (`cell_list` or `batch_cell_list`)
`time_real_us_per_atom`	float	Added by EL; NaN unless component profiling is enabled
`time_reciprocal_us_per_atom`	float	Added by EL; NaN unless component profiling is enabled
`backend_comparable`	bool	Added by NL to mark rows included in backend-comparison plots
`timing_scope`	str	Added by NL to separate scalar backend-comparison rows from coverage-only eager, backend-specific, pair-centric, and cluster-tile rows
`allocation_boundary`	str	Added by NL to identify caller-preallocated, API-managed, or JIT-managed buffers
`derivative_contract`	str	Added by EL; reportable rows use `energy_autograd`
`workload`	str	Added by EL; reportable rows use `energy_forces_charge_gradients`
`compute_forces`	bool	Added by EL; always `True` for the reportable workload
`compute_charge_gradients`	bool	Added by EL; always `True` for the reportable workload
`component_profiled`	bool	Added by EL; `False` for the full-only reportable timing contract
`pme_cache_mode`	str	Added by EL; PME rows use `k_squared_only` or `shared_cell_k_squared`; fixed-cell volume, inverse-cell, and spline-modulus metadata are also precomputed outside timing
`provenance_version`	str	Version of the benchmark provenance schema
`run_id`	UUID	Shared identity for all shards in one reportable run
`gpu_context`	JSON str	Comparable GPU model, compute capability, memory, and driver context
`software_context`	JSON str	Python/framework versions, Git revision, and benchmark-source fingerprint
`execution_context`	JSON str	Per-shard host and physical GPU UUID; may differ across compatible cluster nodes
`runtime_context`	JSON str	Backend runtime and allocator settings, including requested and actual JAX x64/JIT state; must match within each backend
`input_context`	JSON str	Content fingerprints for external NH3 and DFT-D3 inputs; absolute scratch paths are omitted

When Torch and JAX runs share an output directory, each backend rerun replaces only its own rows and preserves the other backend’s rows after validating the run, GPU, software, source, input, and per-backend runtime provenance. An EL rerun with an explicit method also preserves the other method’s rows within that backend, using the same compatibility checks. Compatible scheduler shards may use different hosts and physical GPU UUIDs. Backends contributing to the same CSV must use matching system filters; an asymmetric filter can change the external-input fingerprint and is rejected instead of being silently merged. Failed, skipped, and OOM cases are written directly into the main CSV with success=False; the plotter filters those rows out. The suite no longer writes separate failure files.

The committed H100 CSVs use provenance schema version 2 and the current EL energy-autograd contract. Start a fresh run directory for each collection. Bundled docs validate each complete module/backend/method collection across all systems, cutoffs, and scaling modes, grouping single and batched variants together. Every configured case must occur exactly once. Benchmark collection and external docs overrides retain suite-wide run validation.

Reproducing#

Run from the repository root. Module-specific flags are documented on each module’s doc page; the flags below are common to all three reportable runners. This standardized suite is the benchmark merge gate.

python -m benchmarks.neighborlist.benchmark_neighborlist \
    --config benchmarks/neighborlist/benchmark_config.yaml \
    --output-dir "$BENCHMARK_SCRATCH/results/manual-nl-run"

Swap in benchmarks.interactions.dispersion.benchmark_dftd3 or benchmarks.interactions.electrostatics.benchmark_electrostatics_suite for the other modules, or invoke all three via the unified suite:

bash benchmarks/run_reportable_suite.sh \
    --output-dir "$BENCHMARK_SCRATCH/results/reportable-run"

The wrapper defaults to --backend all: Torch and JAX run for NL, D3, and electrostatics, and the direct Warp pass runs for NL. Use --backend both only when you intentionally want the two framework backends without Warp NL rows.

Use a fresh directory for a new run. Pass --resume only to continue the run already recorded there, or pass the same --run-id to parallel scheduler shards; a mismatched run ID is rejected before execution.

Electrostatics has a separate optional surface: benchmark_electrostatics.py uses benchmark_config_extended.yaml for point-charge/slab/DSF studies and benchmark_config_multipole.yaml for multipoles. It depends on additional benchmark packages not declared in the project extras and does not write the reportable el-*.csv schema. Do not substitute it for benchmark_electrostatics_suite.py in merge-gate commands.

The docs CSVs are reportable benchmark outputs: they use the full configured grid, 3 warmups, and 10 timed runs unless an explicit command-line filter is shown. Reduced smoke runs should write to a separate output directory.

The original H100 CSVs were collected sequentially through the regular Slurm batch queue, with one H100 active at a time. The final rows come from these source-identical jobs:

Job	Accepted scope	Host	Logged wall time
`13573992`	Complete Torch/JAX suite, except three Torch NL shards replaced below	`pool0-01777`	2 h 40 min 35 s
`13574007`	Warp NL, except the two batch-scaling shards replaced below	`pool0-01777`	5 min 10 s
`13582160`	Fixed-affinity replacements for three Torch NL and two Warp NL shards; final validation	`pool0-01787`	3 min 42 s

The contributing jobs used 2 h 49 min 27 s of logged one-GPU time. Collection ran from 06:39:28 to 09:58:00 PDT, a 3 h 18 min 32 s wall span including audit and queue gaps, with peak concurrency of one H100. An intermediate whole-shard rerun (13581772, 1 min 59 s) was fully superseded after the cross-run scaling audit found that timing noise had moved to another point; it is not represented in the published rows. The accepted replacement shards used one fixed logical CPU to remove host scheduling noise. The configuration plans 3,504 rows; the original emitted files contain 3,356. Independent smoothness, endpoint, and prior-run reproducibility checks cover the available measurements.

CSV rows record steady-state per-benchmark timings. Scheduler time also includes compilation, warmup, input loading, process startup, and cleanup.

For the JAX backend, pass --backend jax. Reportable runners preserve explicit JAX/XLA settings, otherwise default the memory fraction to 0.95 while leaving XLA’s default preallocation policy in effect. The unified suite enables x64 before import because its electrostatics pass requires float64. For D3 on offline clusters, pass --d3-params-path to a scratch-local dftd3_parameters.pt file or pre-populate $XDG_CACHE_HOME/nvalchemiops/dftd3_parameters.pt.

Visualization#

Sphinx’s generate_plots hook reads the standardized NL/D3/EL suite CSVs and the separate dynamics CSV schema it knows how to parse, then writes PNGs to ../_static/. The benchmark pages embed those images.