---
orphan: true
---

# Benchmark Results

Pre-computed CSVs consumed by the Sphinx docs build. The shipped numbers
under this directory were produced on an **NVIDIA H100 80 GB HBM3
(Hopper)** and cover three modules (neighbor list, DFT-D3 dispersion,
electrostatics) across two chemical systems (CsCl, NH₃) and three
scaling modes.

The current snapshot was collected on 8 July 2026 under run ID
`b890cd6794884c1f9e9b143e104fa6da` and source fingerprint
`695f537dbf4dc6b6e33911363250f629af2629f489a38c583a27c5370373b770`.
Its 18 reportable CSVs contain all 3,504 planned rows: 3,404 successful
measurements and 100 explicit capacity-limit rows. The failures comprise 88
`OutOfMemoryError` rows, 9 strict-PME `JaxRuntimeError` rows, and 3
`SkippedAfterOOM` rows for JAX Ewald.

Every CSV embeds the same fingerprint in ``software_context``. The measured
source tree was clean at Git head
`66b2aa334a9f2d6b138bf0d1ee87da0e09055593`; later documentation-data edits do
not alter a timed kernel, callable, grid, or CSV value. The software context was
Python 3.13.9, Torch 2.12.0+cu126, JAX/JAXlib 0.9.0.1, Warp 1.13.0, CUDA 12.6,
and ALCHEMI Toolkit-Ops 0.4.0. Collection used NVIDIA H100 80 GB HBM3 GPUs
(compute capability 9.0) with driver 535.216.03. A future kernel, public API,
timing-boundary, or grid change requires a complete replacement run.

See the per-module doc pages for how to read the plots and how to
reproduce:

- `../neighborlist.md`
- `../dftd3.md`
- `../electrostatics.md`

## File naming

Names follow the scheme emitted by
`benchmarks.suite_utils.make_csv_name(module, system, mode)`:

```text
{module}-{system}-{mode-slug}.csv
```

Where `module` ∈ `{nl, d3, el}`, `system` ∈ `{cscl, nh3}`, and
`mode-slug` ∈ `{system-size-scaling, constant-workload-scaling,
batch-scaling}`. Example: `nl-cscl-system-size-scaling.csv`.

The current reportable NL/D3/EL suite uses only root-level `nl-*.csv`,
`d3-*.csv`, and `el-*.csv` files. Separate dynamics and segment-operation CSVs
may also remain at the root for their own docs pages. Outputs from the optional
extended electrostatics runner use `electrostatics_benchmark_*.csv` names and a
different schema; they are not read by the reportable plot generation path.

## CSV schema

Emitted by `benchmarks.suite_utils.build_result`:

| Column | Type | Description |
|---|---|---|
| `system` | str | `cscl` or `nh3` |
| `scaling_mode` | str | `system_size`, `constant_workload`, or `batch_scaling` |
| `method` | str | NL strategy (`naive_scalar`, `naive_tile`, `cell_list_atom_centric`, `cell_list_pair_centric`, `cluster_tile`, plus batch-prefixed concrete APIs where applicable), `dftd3` (D3), or `pme` / `ewald` (EL) |
| `backend` | str | `torch`, `jax`, or `warp` where supported |
| `atoms_per_system` | int | Atoms in one system |
| `batch_size` | int | Number of systems in the batch |
| `total_atoms` | int | `atoms_per_system` × `batch_size` |
| `time_us_per_atom` | float | Mean μs per atom across the batch timing |
| `throughput_atoms_per_sec` | float | Derived throughput |
| `mem_delta_mb` | float | Torch CUDA allocator delta from the pre-timing measurement call (MB); NaN for JAX |
| `mem_peak_gb` | float | Torch CUDA allocator peak (GB); NaN for JAX |
| `timing_runs` | int | Number of timed calls represented by the row |
| `warmup_runs` | int | Number of untimed warmup calls before measurement |
| `timing_method` | str | Timing path used for the row, such as `torch_cuda_events`, `jax_wall_block_until_ready`, or the serial fallback `jax_wall_block_each` |
| `timing_method_real` | str | Added by EL; `not_measured` unless component profiling is enabled |
| `timing_method_reciprocal` | str | Added by EL; `not_measured` unless component profiling is enabled |
| `compile_policy` | str | Compile/warmup policy; shipped rows use `warmup_excluded` |
| `success` | bool | `False` rows are filtered by the plotter |
| `error` | str | Concise failure or skip message for `success=False` rows |
| `error_type` | str | Stable failure class, such as `OutOfMemoryError`, `JaxRuntimeError`, `UnsupportedConfiguration`, `SkippedByPolicy`, or `SkippedAfterOOM` |
| `failure_stage` | str | Optional; populated by failure paths that can identify the setup or timing stage that raised the error |
| `cutoff` | float | Added by NL and D3 |
| `configured_max_neighbors` | int | Added by NL; neighbor-matrix capacity selected before the measured call |
| `max_neighbors` | int | Added by NL; width of the allocated or returned dense neighbor matrix |
| `max_total_cells` | int | Added by cell-list NL rows; allocated cell-grid capacity |
| `cell_list_min_cells` | int | Added by NL backends that expose this cell-grid metadata; minimum per-axis cell count used by the selected strategy |
| `total_neighbor_pairs` | int | Added by NL; backend-specific pair/storage count (dense paths report allocated slots, while direct Warp rows report populated pairs) |
| `accuracy` | float | Added by EL |
| `time_d3_us_per_atom` | float | Added by D3 (excludes NL build time) |
| `neighbor_setup_method` | str | Added by D3; neighbor-list setup API used before timing (`cell_list` or `batch_cell_list`) |
| `time_real_us_per_atom` | float | Added by EL; NaN unless component profiling is enabled |
| `time_reciprocal_us_per_atom` | float | Added by EL; NaN unless component profiling is enabled |
| `backend_comparable` | bool | Added by NL to mark rows included in backend-comparison plots |
| `timing_scope` | str | Added by NL to separate scalar backend-comparison rows from coverage-only eager, backend-specific, pair-centric, and cluster-tile rows |
| `allocation_boundary` | str | Added by NL to identify caller-preallocated, API-managed, or JIT-managed buffers |
| `derivative_contract` | str | Added by EL; reportable rows use `energy_autograd` |
| `workload` | str | Added by EL; reportable rows use `energy_forces_charge_gradients` |
| `compute_forces` | bool | Added by EL; always `True` for the reportable workload |
| `compute_charge_gradients` | bool | Added by EL; always `True` for the reportable workload |
| `component_profiled` | bool | Added by EL; `False` for the full-only reportable timing contract |
| `pme_cache_mode` | str | Added by EL; PME rows use `full_static` when fixed-cell volume, inverse-cell, and spline-modulus metadata are precomputed outside timing |
| `provenance_version` | str | Version of the benchmark provenance schema |
| `run_id` | UUID | Shared identity for all shards in one reportable run |
| `gpu_context` | JSON str | Comparable GPU model, compute capability, memory, and driver context |
| `software_context` | JSON str | Python/framework versions, Git revision, and benchmark-source fingerprint |
| `execution_context` | JSON str | Per-shard host and physical GPU UUID; may differ across compatible cluster nodes |
| `runtime_context` | JSON str | Backend runtime and allocator settings, including requested and actual JAX x64/JIT state; must match within each backend |
| `input_context` | JSON str | Content fingerprints for external NH3 and DFT-D3 inputs; absolute scratch paths are omitted |

When Torch and JAX runs share an output directory, each backend rerun replaces
only its own rows and preserves the other backend's rows after validating the
run, GPU, software, source, input, and per-backend runtime provenance.
Compatible scheduler shards may use different hosts and physical GPU UUIDs.
Backends contributing to the same CSV must use matching system filters; an
asymmetric filter can change the external-input fingerprint and is rejected
instead of being silently merged.
Failed, skipped, and OOM cases are
written directly into the main CSV with `success=False`; the plotter filters
those rows out. The suite no longer writes separate failure files.

The committed H100 CSVs use provenance schema version 2 and the current EL
energy-autograd contract. All 18 files share the run ID shown above; start a
fresh run directory and publish a complete replacement set for future reruns.

## Reproducing

Run from the repository root. Module-specific flags are documented on
each module's doc page; the flags below are common to all three reportable
runners. This standardized suite is the benchmark merge gate.

```bash
python -m benchmarks.neighborlist.benchmark_neighborlist \
    --config benchmarks/neighborlist/benchmark_config.yaml \
    --output-dir "$BENCHMARK_SCRATCH/results/manual-nl-run"
```

Swap in `benchmarks.interactions.dispersion.benchmark_dftd3` or
`benchmarks.interactions.electrostatics.benchmark_electrostatics_suite` for
the other modules, or invoke all three via the unified suite:

```bash
bash benchmarks/run_reportable_suite.sh \
    --output-dir "$BENCHMARK_SCRATCH/results/reportable-run"
```

The wrapper defaults to ``--backend all``: Torch and JAX run for NL, D3, and
electrostatics, and the direct Warp pass runs for NL. Use ``--backend both``
only when you intentionally want the two framework backends without Warp NL
rows.

Use a fresh directory for a new run. Pass `--resume` only to continue the run
already recorded there, or pass the same `--run-id` to parallel scheduler
shards; a mismatched run ID is rejected before execution.

Electrostatics has a separate optional surface:
`benchmark_electrostatics.py` uses `benchmark_config_extended.yaml` for
point-charge/slab/DSF studies and `benchmark_config_multipole.yaml` for
multipoles. It depends on additional benchmark packages not declared in the
project extras and does not write the reportable `el-*.csv` schema. Do not
substitute it for `benchmark_electrostatics_suite.py` in merge-gate commands.

The docs CSVs are reportable benchmark outputs: they use the full configured
grid, 3 warmups, and 10 timed runs unless an explicit command-line filter is
shown. Reduced smoke runs should write to a separate output directory.

The shipped H100 CSVs were collected sequentially through the regular Slurm
``batch`` queue, with one H100 active at a time. The final rows come from these
source-identical jobs:

| Job | Accepted scope | Host | Logged wall time |
|---|---|---|---:|
| `13573992` | Complete Torch/JAX suite, except three Torch NL shards replaced below | `pool0-01777` | 2 h 40 min 35 s |
| `13574007` | Warp NL, except the two batch-scaling shards replaced below | `pool0-01777` | 5 min 10 s |
| `13582160` | Fixed-affinity replacements for three Torch NL and two Warp NL shards; final validation | `pool0-01787` | 3 min 42 s |

The contributing jobs used 2 h 49 min 27 s of logged one-GPU time. Collection
ran from 06:39:28 to 09:58:00 PDT, a 3 h 18 min 32 s wall span including audit
and queue gaps, with peak concurrency of one H100. An intermediate whole-shard
rerun (`13581772`, 1 min 59 s) was fully superseded after the cross-run scaling
audit found that timing noise had moved to another point; it is not represented
in the published rows. The accepted replacement shards used one fixed logical
CPU to remove host scheduling noise. Final validation reported 3,504 planned
and emitted rows, and independent smoothness, endpoint, and prior-run
reproducibility checks found no remaining timing outliers.

CSV rows record steady-state per-benchmark timings. Scheduler time also includes
compilation, warmup, input loading, process startup, and cleanup.

For the JAX backend, pass `--backend jax`. Reportable runners preserve explicit
JAX/XLA settings, otherwise default the memory fraction to `0.95` while leaving
XLA's default preallocation policy in effect. The unified suite enables x64
before import because its electrostatics pass requires float64.
For D3 on offline clusters, pass `--d3-params-path` to a scratch-local
`dftd3_parameters.pt` file or pre-populate
`$XDG_CACHE_HOME/nvalchemiops/dftd3_parameters.pt`.

## Visualization

Sphinx's generate_plots hook reads the standardized NL/D3/EL suite CSVs and
the separate dynamics CSV schema it knows how to parse, then writes PNGs to
`../_static/`. The benchmark pages embed those images.