Crystal structure prediction#

Crystal structure prediction (CSP) explores possible arrangements of molecules in a crystal. The nvalchemi.csp package provides four parts of this search:

  • Prepare the search: assemble molecular conformers into a formula unit, estimate starting cell volumes, and choose compatible space groups. The molecule helpers use RDKit when available; raw tensor inputs are also supported.

  • Generate starting structures: OverlapReliefPacker chooses input conformers and adjusts rigid molecular placements and cells to reduce overlaps. CSPGenerator also accepts user-defined packers that return rigid ASU data or explicit atomistic coordinates in a Batch.

  • Save and expand structures: the rigid packer returns an asymmetric-unit (ASU) representation containing independent molecule placements, a cell, and symmetry information. Zarr storage retains these data without writing every atom in the cell. Selected structures can be expanded to full-cell Toolkit Batch objects for physical optimization. Here “compact” describes the storage representation, regardless of cell volume.

  • Screen similar structures: RadialComparisonIndex compares local atom environments to find possible duplicates after optimization. It also accepts fully periodic, partially periodic, and nonperiodic structures from other sources. See comparison for its limits and examples.

Packing does not rank crystals

Packing produces starting geometries with intermolecular clashes resolved to the specified overlap tolerance. Physical energies and polymorph ranking require a suitable model and further optimization.

For API signatures and parameter details, see the CSP API reference. See also the AtomicData and Batch for Toolkit batches and the Dynamics: Optimization and Molecular Dynamics for optimization with a physical model.

Generate starting structures#

Prepare water with the optional RDKit helpers, request 200 structures on one GPU, and expand two accepted structures. Install nvalchemi-toolkit[rdkit] and use an environment with CUDA support.

Note

The \((1000\text{-}1200)\,\mathrm{\AA}^3\) starting cells are deliberately oversized for one water molecule so the example completes quickly. They are unsuitable starting volumes for an ice search.

import torch

from nvalchemi.csp import SpaceGroupPolicy
from nvalchemi.csp.chem import (
    RDKitConformerConfig,
    build_molecular_packing_input,
    generate_conformers_from_smiles,
)
from nvalchemi.csp.packer import OverlapReliefPacker, OverlapReliefConfig

# Generate one water conformer with RDKit.
molecule = generate_conformers_from_smiles(
    "O", config=RDKitConformerConfig(num_conformers=1, random_seed=7, num_threads=1)
)
# Assemble the molecules and contact distances for one formula unit.
packing_input = build_molecular_packing_input([molecule])
# Configure the starting cells, symmetry, and finite trial budget.
config = OverlapReliefConfig(
    z=1,
    z_prime=1,
    batch_size=64,
    max_candidates=1000,
    cell_volume_range=(1000.0, 1200.0),
    space_groups=SpaceGroupPolicy.fixed(1),
)
# Bind the packer to the selected GPU.
packer = OverlapReliefPacker(
    config=config,
    device="cuda:0",
)
# Generate accepted starting structures and inspect any shortfall.
result = packer(
    packing_input,
    num_samples=200,
    rng=torch.Generator(device="cuda:0").manual_seed(7),
    run_id=77,
)
print("accepted:", len(result), "trials:", result.generated_count)
print("stopped:", result.stop_reason)
if len(result) < 2:
    raise RuntimeError("Fewer than two candidates were accepted")

# Expand two accepted ASU structures into full-cell atomistic data.
selected = result.structures.to_batch(indices=torch.tensor([0, 1]))
assert selected.num_graphs == 2

num_samples requests accepted structures. batch_size limits the trials active at once. By default, max_candidates="auto" limits all initialized trials to 1000 * num_samples for each call. A positive integer sets a fixed budget; explicit None leaves the trial budget unlimited. Rejected trials consume that budget, so a finite search may return fewer structures than requested. Inspect the accepted count and stop reason before continuing.

selected contains full-cell atom coordinates for two candidates.

The automatic cap is a safety limit, not an acceptance guarantee or a wall-clock timeout. At an acceptance rate of about 0.1%, this budget yields the requested count on average and can still return a shortfall. A bounded pre-flight remains useful before scaling up.

Pre-flight large packing searches

With explicit max_candidates=None, the trial budget is unlimited. Unreachable acceptance criteria can keep the search running without useful progress. Before a large or unlimited search, run a short pre-flight with a finite candidate budget, using the intended starting-volume range and per-candidate relaxation limit. Inspect accepted structures versus generated trials. An overly small starting volume can produce low or zero acceptance, and packing does not expand the cell volume. Revisit the volume settings and repeat the pre-flight before scaling up.

preflight = packer(
    packing_input,
    num_samples=32,
    max_candidates=32000,
    run_id=76,
)
print(
    "pre-flight accepted:", len(preflight),
    "trials:", preflight.generated_count,
    "stopped:", preflight.stop_reason,
)

This pre-flight permits at most 32000 initialized trials and retains the configured per-candidate relaxation limit. Use the settings intended for the search. Accepted and generated counts are diagnostic: there is no universal acceptance threshold or automatic decision to continue. Restrictive settings can prevent cell initialization and raise an error. These controls bound work rather than impose a wall-clock timeout.

Distribute candidate generation#

CSPGenerator allocates independent packing work across an existing process group. The application owns process launch and group lifetime. Each process supplies the same prepared formula-unit input; scientific settings such as Z, Z', and space-group policy may differ between ranks. Gloo supports CPU packers; NCCL supports CUDA packers, usually with one process per GPU.

Before packing, the generator automatically compares sha256 across processes to check that they received the same prepared input. The comparison includes conformer coordinates, contact distances, metadata, and the formula-unit volume estimate. It requires exact data: even small coordinate differences cause a mismatch.

packer.device selects where packing runs; the configuration describes the scientific search settings. CSPGenerator takes its execution device from the packer. Each rank can use the same settings on its own device. An input Batch or a downstream model does not select the CSP generator’s device.

This fragment assumes a torchrun launch and an initialized NCCL group. Prepare the input and config on each process:

import os
import torch.distributed as dist
from nvalchemi.csp import CSPGenerator

local_rank = int(os.environ["LOCAL_RANK"])
device = f"cuda:{local_rank}"
rank_packer = OverlapReliefPacker(config=config, device=device)
generator = CSPGenerator(
    rank_packer,
    process_group=dist.group.WORLD,
    gather_to_rank=0,
    expand=False,
)
rank_result = generator(
    packing_input,
    num_samples=200,
    rng=torch.Generator(device=device).manual_seed(7),
    run_id=78,
)
if dist.get_rank() == 0:
    assert rank_result is not None
else:
    assert rank_result is None

num_samples is the global accepted-output target. Default quotas divide it by quotient and remainder. For budget-capable packers, the driver resolves the global candidate cap before allocating it. Custom rank_targets must sum to the positive global target. When rank_candidate_budgets is omitted, each rank gets a share of a finite cap proportional to its target. Shares are rounded down; leftover trials go to the largest fractional shares, with lower group-local ranks winning ties. A zero-target rank receives zero trials. For example, targets [0, 2] receive budgets [0, 2000] under automatic mode. Explicit budgets take precedence and must sum to the global cap. Default targets retain quotient/remainder budget allocation; unlimited caps remain unlimited on each rank. Unused budgets are not redistributed. Packers without candidate budgets need no trial counting and reject explicit rank-budget vectors.

gather_to_rank uses group-local ranks. With a destination, the native result contains rank-major structures and every rank’s completion report. Without a destination, each rank keeps its local result. Gathering requires one native representation and matching property or Batch schemas on all ranks, including empty results; mixing rigid and atomistic output in one gather is rejected.

Ranks agree on the input, execution contract, and run ID at startup, then pack locally to completion. Rank-local output has no terminal rendezvous: errors and completion are local. Gathered output exchanges terminal status before payload transfer and assembly status afterward. Early, failed, and zero-quota ranks may wait for the slowest peer. Set the process-group timeout to cover that skew; unlimited searches may never finish. There is no periodic coordination or peer cancellation. Hard process/device failures and transport failures are outside ordinary Python error agreement.

The generator scopes NCCL communication to the packer’s CUDA device and restores the caller’s current device. Constructor/session failures and collectives inside user callbacks remain caller-owned. Later distributed optimization is a separate Toolkit workflow (see Dynamics: Optimization and Molecular Dynamics).

Distributed setup remains caller-owned. Internally, each grouped generation call uses ProcessGroupContext for the supplied group’s topology and communication device, and collective_error_sync() for startup, packing metadata, and assembly error agreement. Rank-local calls exchange startup metadata and broadcast the run ID, then finish independently. Gathered calls also exchange packing metadata before typed transfers and assembly status afterward. Callbacks, expansion, and AFTER_GENERATE run after these phases; their exceptions remain local.

As an alternative to external PyTorch initialization, use a named Manager group. This setup starts before a process group has been initialized and reuses config from the water example above:

import torch.distributed as dist
from nvalchemi.csp import CSPGenerator
from nvalchemi.csp.packer import OverlapReliefPacker
from nvalchemi.distributed import DistributedManager

DistributedManager.initialize()
manager = DistributedManager()
process_group = None
if dist.is_initialized():
    DistributedManager.create_process_subgroup("packing", size=manager.world_size)
    process_group = manager.group("packing")

packer = OverlapReliefPacker(config, device=manager.device)
generator = CSPGenerator(
    packer,
    process_group=process_group,
    gather_to_rank=0 if process_group is not None else None,
)

manager.group() returns None for the default group, while CSPGenerator interprets None as local execution. Select dist.group.WORLD explicitly or pass an actual named group. Manager initialization in a single-process run may leave PyTorch distributed uninitialized; the example selects local CSP execution in that case. The caller keeps the group alive until generation and any later communication finish. See Distributed runtime utilities for shared runtime helpers and their ordinary-failure boundaries.

Store and relax selected candidates#

The examples below continue from the single-GPU result. For a distributed run, save on the gather destination using its non-None rank_result.

RigidMoleculeASUBatch stores independent molecule placements, shared conformers, cell, and space-group information instead of all full-cell atom coordinates. Save accepted ASU representations first, then expand only the candidates chosen for expensive physical optimization.

from nvalchemi.csp import RigidMoleculeASUZarrReader, RigidMoleculeASUZarrWriter

with RigidMoleculeASUZarrWriter("water-candidates.zarr") as writer:
    writer.write(result.structures)

with RigidMoleculeASUZarrReader("water-candidates.zarr") as reader:
    asu_rows = reader.read(indices=torch.tensor([0, 1]))
    batch = reader.read_batch(indices=torch.tensor([0, 1]), device="cuda:0")
assert asu_rows.num_structures == batch.num_graphs == 2

write() creates a new store and refuses to replace an existing path. Use a fresh path when rerunning the example. append() adds compatible rows; an identical active structure ID is skipped, while conflicting data for that ID raises an error. read_batch() requires explicit row indices so a large store is not expanded accidentally. An empty OverlapReliefPacker result is a valid ASU batch, but handle it before passing data to an optimizer.

Check externally prepared compact data#

Before expanding compact data prepared by another program, call check_integrity() explicitly on the selected structures:

with RigidMoleculeASUZarrReader("water-candidates.zarr") as reader:
    compact = reader.read(indices=torch.tensor([0, 1]), device="cpu")
    compact.check_integrity()
    batch = compact.to_batch(device="cuda:0")

Let F be the number of molecules in one formula unit and Q the total ASU molecule rows. structure_molecule_ptr must start at zero, never decrease, end at Q, and assign F * z_prime rows to each structure. z and z_prime must be positive, z_prime must divide z, and the space group must be in 1-230 with z / z_prime bundled operations. Rows repeat the formula-molecule order for each independent copy; every conformer index must belong to that molecule’s pool. The checker verifies these index relationships and raises ValueError when they fail.

Geometry still needs to be valid: fractional centers must be finite, but may lie outside [0, 1). Cartesian rotations must be finite proper rotations, applied as local_positions @ rotation.T. Cells must be finite nonsingular row-vector lattices in angstroms with positive volume; the selected fractional symmetry operations must preserve the cell metric. Index checking does not certify these conditions. Invalid geometry can produce incorrect or nonfinite expanded coordinates even when index checking passes.

Construction checks tensor shapes, dtypes, device consistency, and property names. Construction, storage reads, packing, and expansion do not automatically run the index checker. Skipping it on malformed data can select another structure’s rows or another group’s operations. Storage-layout checks do not establish these relationships or geometry validity. Checking on CPU before expansion avoids additional GPU readbacks; checking device-resident data may synchronize. The checker scans integer metadata in proportion to the selected data, and its overhead has not been measured.

The expanded batch can enter Toolkit dynamics. This one-step demonstration shows the handoff; DemoModel is not a physical crystal potential and its output has no CSP ranking value. For a search, use a model suitable for the composition and intermolecular interactions, configure its required neighbor data, and relax to an appropriate convergence criterion (see Dynamics: Optimization and Molecular Dynamics).

from nvalchemi.dynamics import FIRE2
from nvalchemi.models.demo import DemoModel, DemoModelWrapper

demo_model = DemoModelWrapper(DemoModel()).to("cuda:0")
with FIRE2(model=demo_model, dt=0.05, n_steps=1) as optimizer:
    demo_relaxed = optimizer.run(batch)

Space-group symmetry during optimization

Atomistic optimization does not enforce the generated space group. The stored space-group field records the source packing; it does not describe the symmetry of the optimized coordinates. See source metadata for the retained provenance fields.

In a single-process run, each accepted structure has a stable [run_id, accepted_structure_number] ID, with the second number starting at zero. Distributed runs use rank-strided numbers to keep IDs unique. Expansion carries these IDs and the source space group into the atomistic Batch.

Run packing in a generation pipeline#

CSPGenerator subclasses AtomisticGenerator and reuses its RNG, conditioning, hooks, sessions, streaming, and pipeline lifecycle. By default it expands rigid ASU results to full-cell Batch objects; native Batch results pass through unchanged. This example reuses packer, packing_input, and demo_model above:

from nvalchemi.csp import CSPGenerator

generator = CSPGenerator(packer, num_samples=4, seed=79)
pipeline = generator | FIRE2(model=demo_model, dt=0.05, n_steps=1)
with pipeline:
    relaxed = pipeline(packing_input)

A finite budget may produce fewer structures than requested. An empty Batch skips downstream stages. AFTER_GENERATE hooks see the atomistic output; native results and their reports describe packing before those hooks filter or modify it. Collective dynamics need a separate integration when ranks have unequal outputs or gathered non-destination ranks have no output.

Use on_result to inspect or persist native results before expansion, hooks, or dynamics. For rigid results, the existing ASU writer can persist candidates even if later optimization fails:

def make_persistence_callback(writer):
    initialized = False

    def persist(result):
        nonlocal initialized
        if initialized:
            writer.append(result.structures)
        else:
            writer.write(result.structures)
            initialized = True

    return persist


with RigidMoleculeASUZarrWriter("pipeline-candidates.zarr") as writer:
    generator = CSPGenerator(
        packer,
        num_samples=4,
        seed=79,
        on_result=make_persistence_callback(writer),
    )
    with generator | FIRE2(model=demo_model, dt=0.05, n_steps=1) as pipeline:
        relaxed = pipeline(packing_input)

Use a fresh destination owned by this callback. The first successful call creates the store with write(); later calls use append(). An empty first result also creates a valid store. Later results must have the same formula input and compatible property names, dtypes, and trailing dimensions. Completed writes survive a later optimization failure.

An existing path raises FileExistsError; this callback does not resume an existing search or overwrite data. Its state belongs to this writer and callback lifetime. Failed creation leaves the flag unset, but partial I/O may leave an existing path, so retry and interrupted-write recovery are not guaranteed. It provides no concurrent-writer coordination.

The callback runs once per result owner, including empty results: each rank in rank-local mode or only the destination in gather mode, after communication completes. Callback exceptions are local and callbacks must not require a peer collective. Native Batch payloads alias downstream output, so copy the payload explicitly to retain an unchanged snapshot. ASU storage remains for rigid representations; this change does not make the ASU writer a Batch store.

expand=False returns a raw PackingResult, or None on non-destination ranks in gather mode. Raw results bypass Batch hooks and cannot feed dynamics. Expanded non-destination ranks receive an empty Batch. Driver compilation is unsupported: CSPGenerator.compile() and pipeline compilation through a CSP stage raise; compiled scientific internals inside a packer remain possible.

Save a generation recipe#

Local CSP generators can be saved as JSON construction recipes. The recipe includes the packer’s configuration and device, generator options, output declaration, and supported hooks and callbacks. Loading reconstructs the packer and CSP generator.

A pipeline can be saved only when every stage supports serialization. Save the CSP generator separately here, then construct the dynamics stage after loading:

from nvalchemi.csp import CSPGenerator

recipe_generator = CSPGenerator(packer, num_samples=4, seed=79)
recipe = recipe_generator.model_dump_json()
restored_generator = CSPGenerator.model_validate_json(recipe)

with restored_generator | FIRE2(model=demo_model, dt=0.05, n_steps=1) as restored_pipeline:
    relaxed = restored_pipeline(packing_input)

A restored recipe starts a fresh generation session. It preserves the configured seed, but not a consumed random-number state, call counters, or run identifiers. Molecular inputs and per-call options are supplied again when running the restored pipeline.

Custom packers must provide to_spec() returning a BaseSpec to support serialization. Callback functions must be importable, or callable objects must provide their own construction spec. Ordinary instance-bound methods are not serializable by import path alone. A live distributed process group cannot be saved in this recipe; its lifetime and binding remain caller-owned.

Supply another packing algorithm#

Implement CrystalPacker structurally: expose device and a local pack method; no Toolkit inheritance or common configuration base is required. The driver supplies a PackingContext for identity and never supplies a process group to the algorithm. A zero-target call returns a typed empty result with the same schema as a positive call and consumes no sampling RNG.

The following small example repeats supplied geometry to demonstrate the atomistic result contract. It does not search for packings:

import secrets
import torch
from nvalchemi.csp.packer import PackingContext, PackingReport, PackingResult
from nvalchemi.data import Batch


class FixedGeometryPacker:
    def __init__(self, template):
        self.device = template.device
        self.template = template.clone()
        self.template.add_system_property(
            "csp_source_structure_id",
            torch.zeros((1, 2), dtype=torch.int64, device=self.device),
        )

    def pack(
        self, inputs, *, num_samples=1, rng=None, context=None, **options
    ):
        if options:
            raise TypeError(f"Unknown packing options: {sorted(options)}")
        context = context or PackingContext(secrets.randbits(63))
        ids = context.structure_ids(num_samples, device=self.device)
        if num_samples == 0:
            batch = Batch.empty(
                num_systems=0, num_nodes=0, num_edges=0,
                template=self.template, device=self.device,
            )
        else:
            rows = []
            for ordinal in range(num_samples):
                row = self.template.clone()
                row["csp_source_structure_id"] = ids[ordinal:ordinal + 1]
                rows.append(row)
            batch = Batch.from_data_list(rows, device=self.device)
        report = PackingReport(
            context.rank, num_samples, num_samples, "target_reached"
        )
        return PackingResult(batch, context.run_id, (report,))

Supply an AtomicData template with positions, atomic numbers, cell, and PBC, then use CSPGenerator(FixedGeometryPacker(template)). Scientific packers also provide any fields required by subsequent dynamics. Native Batch results carry system-level csp_source_structure_id as int64 [P, 2]; rigid packers use the same IDs in RigidMoleculeASUBatch.structure_ids. IDs remain [run_id, rank + world_size * local_accepted_ordinal], independent of sampling RNG.

PackingResult exposes requested_count, accepted_count, complete, and ordered reports. generated_count is None when an algorithm does not count candidates. Incomplete reports may use algorithm-specific stop reasons; gathered results preserve these rather than attributing every shortfall to candidate-budget exhaustion. A rigid result must retain equivalent formula data, including conformer coordinates. Algorithms that change internal geometry beyond that library return Batch.

Candidate-budget support is optional. Implement resolve_candidate_budget(num_samples=..., **pack_options) without sampling and accept an allocated candidate_budget in pack when that capability is useful. A zero allocated cap produces an empty result without sampling. The scientific candidate loop stays in the algorithm; overlap relief retains its private resumable core. Public rounds and a shared candidate controller are deferred until a second implementation demonstrates a common lifecycle. These choices follow the extensibility goal of the API design discussion while supporting both rigid and explicit-coordinate output.

Optional deduplication for custom packers#

Custom packers, including those based on generative models, may propose repeated crystal candidates. DeduplicateHook provides optional filtering of their generated structures before downstream processing.

The bundled radial engine compares expanded atomic structures within each generated Batch using approximate similarity. It keeps no reference set between calls.

The following example uses FixedGeometryPacker from above, which deliberately repeats the supplied geometry:

from nvalchemi.csp.hooks import DeduplicateHook

custom_packer = FixedGeometryPacker(template)
generator = CSPGenerator(
    custom_packer,
    num_samples=4,
    seed=79,
    hooks=[DeduplicateHook.radial(cutoff=5.0, threshold=0.05)],
)
with generator | FIRE2(model=demo_model, dt=0.05, n_steps=1) as pipeline:
    relaxed = pipeline(packing_input)

Choose the cutoff and threshold for your application. See radial() for confirmation options and DeduplicationEngine for custom engines.

Screen similar and duplicate structures#

A CSP campaign can generate and optimize \(10^4\) to \(10^6\) trial structures, many of which may converge to similar geometries. A pool of \(N\) structures has \(N(N-1)/2\) possible unordered pairs: about 50 million for \(10^4\) structures and 500 billion for \(10^6\). The comparison API makes each first-pass comparison fast; exhaustive pair enumeration still grows quadratically. For very large pools, select plausible pairs or work in chunks.

What the radial mismatch score means#

For each atom, the index sorts distances to neighbors within cutoff, in \(\mathrm{\AA}\); shorter lists are padded with the cutoff distance. It finds the most similar compatible atom environment in the other structure by comparing those distance patterns, regardless of the atoms’ coordinates in a common frame. Every atom environment must find a match in the other structure, and the comparison runs in both directions. Several atoms may match the same environment: this is not a one-to-one atom assignment.

The radial mismatch score is the largest relative mismatch remaining after these matches. For two positive distances, the mismatch is max(d1, d2) / min(d1, d2) - 1. At threshold=0.1, the larger distance may be at most 10% above the smaller one. For example, \(2.0\,\mathrm{\AA}\) and \(2.1\,\mathrm{\AA}\) have a score of 0.05. The threshold is dimensionless, not a tolerance in \(\mathrm{\AA}\). Lower scores mean closer local distance patterns.

The score ignores angles, chirality, energy, and distances beyond cutoff. Distinct crystals can therefore have low mismatch scores: matches can be false positives. For molecular crystals, OXtalign compares molecular packing; pymatgen’s StructureMatcher offers general crystal-structure matching. These are external options for confirming important matches. With deduplicate(confirm=...), the caller supplies a more detailed comparison function. CSP passes it the pairs that survive the radial screen and uses the confirmed subset to choose retained representatives. No final crystallographic comparator is bundled here. The index honors fully periodic, partially periodic, or nonperiodic coordinates according to the Batch cell and PBC flags. Periodic directions require both a cell and PBC flags; a cell alone does not enable periodic comparison.

Choose how atom types restrict matches#

Each distance list belongs to a central atom and describes its neighbors. An untyped first screen lets any central atoms match based on distances alone; it avoids type preparation and uses the simplest descriptor, but can leave more false positives. Supplying atom_types restricts which central atoms may match. Setting typed_neighbors=True also separates their neighbor distances by atom type. Callers can choose the level of chemical selectivity:

Settings

Which atoms may match?

atom_types=None

Any central atoms; elements are not inferred.

atom_types=batch.atomic_numbers, typed_neighbors=False

Central atoms of the same element; all neighbor distances remain in one list.

Topological atom_types, typed_neighbors=False

Central atoms with the same molecular connectivity role; neighbor distances remain in one list.

Types with typed_neighbors=True

Central atoms of the same type, with neighbor distances compared separately by neighbor type.

Atomic numbers alone do not identify an atom’s role within a molecule. In ethanol, both the methyl and methylene carbons have atomic number 6 but different bond connections. Topological atom types distinguish those roles. With explicit hydrogens, the three methyl hydrogens share one type, the two methylene hydrogens another, and the hydroxyl hydrogen has its own. The helpers group atoms that can be exchanged by relabeling a molecule while preserving elements and bond connections.

Topological type numbers are local labels. Build one TopologicalAtomTypeMap from the formula unit’s ordered atomic numbers and bonds, then use it to label both relaxed candidates and any experimental structures compared with them. The labels ignore bond order, charge, isotope labels, stereochemistry, geometry, and implicit hydrogens. For a single graph, use topological_atom_types_from_connectivity() with tensors or topological_atom_types_from_mol() with RDKit. The default typed_neighbors=True separates neighbor lists when types are supplied. Callers can run successive typing screens on surviving pairs; the Toolkit does not perform that sequence automatically.

Deduplicate a relaxed structure pool#

After the generate -> optimize workflow above, collect relaxed structures generated from the same molecular input into a standard Toolkit Batch, called batch below. Assume optimization has preserved molecular connectivity: the structures may have different geometries, but they share the same molecular topology. Retain the molecular-bond matrix as template_adjacency, using the same atom order as packing_input.atomic_numbers. ASU expansion records each atom’s source index, which the type map uses to label relaxed structures even when they have different Z or Z'. Keep that index aligned with its atom during optimization and storage. The map checks indices and elements, but cannot establish that a loaded Batch came from this particular formula unit. See Source metadata after geometry changes for the provenance field.

Topological types are optional for approximate matching. An index without types compares local distances regardless of atom identity; using atomic numbers as types restricts central atoms to the same element. Either may leave more false-positive matches than topology-based screening, because atoms of the same element can play different roles in a molecule.

For a pool with a manageable number of possible pairs, refine matches in three steps: distances alone, then the molecular role of the central atom, then the roles of both the central atom and its neighbors. Each step keeps only pairs that passed the preceding one. Release each index before building the next. Replace the ... placeholders with your formula-unit input, its bond matrix, and a relaxed Toolkit Batch from the same search:

from nvalchemi.csp.chem import TopologicalAtomTypeMap
from nvalchemi.csp.comparison import RadialComparisonIndex

packing_input = ...  # Formula-unit input used to generate this search pool.
template_adjacency = ...  # Boolean bonds in packing_input.atomic_numbers order.
batch = ...  # Toolkit Batch of optimized trial crystals with ASU source fields.

cutoff = 15.0  # angstroms
threshold = 0.05  # maximum relative distance mismatch

# 1. Compare local distances without atom types.
untyped_index = RadialComparisonIndex.build(batch, cutoff=cutoff)
pairs = untyped_index.find_matches(threshold=threshold)
del untyped_index

# 2. For pairs marked as possible duplicates by step 1, require the
#    central atoms to have the same molecular role.
type_map = TopologicalAtomTypeMap(
    packing_input.atomic_numbers, template_adjacency
)
topology_types = type_map.for_batch(batch)
center_index = RadialComparisonIndex.build(
    batch, cutoff=cutoff, atom_types=topology_types, typed_neighbors=False
)
pairs = center_index.find_matches(threshold=threshold, pair_indices=pairs)
del center_index

# 3. For pairs marked as possible duplicates by step 2, also compare
#    neighbors by their molecular roles.
typed_index = RadialComparisonIndex.build(
    batch, cutoff=cutoff, atom_types=topology_types, typed_neighbors=True
)
approx_duplicates = typed_index.find_matches(
    threshold=threshold, pair_indices=pairs
)
approx_scores = typed_index.score_pairs(approx_duplicates)

Each row of approx_duplicates identifies two structures with similar local distance patterns; approx_scores is aligned with those rows. These are approximate matches, not confirmed duplicate crystals. This example collects the pairs from each stage, so the pair tensor itself can grow large. If you only need representatives, build just the typed_index from step 3 and skip the first two indices and pair lists:

result = typed_index.deduplicate(threshold=threshold)
unique_indices = result.retained_indices
deduplicated_batch = batch[unique_indices]

deduplicated_batch contains the provisional representatives; the other structures are assigned to them in result.representative_indices. If you need confirmed structural equivalence before discarding a candidate, pass a more specific comparison as deduplicate(confirm=...). The callback receives proposed pairs and returns only confirmed ones in their original pair order. Confirming after deduplication cannot recover a structure already skipped by its greedy pass. Within each typed screen, type IDs must have the same meaning for every structure.

deduplicate() is greedy: it visits structures in input order and assigns each candidate to the first earlier retained representative that matches it. A match at the chosen threshold is not necessarily transitive: A may match B and B may match C even though A does not match C. In the order A, B, C, the algorithm retains A, assigns B to A, then retains C because it compares C with A, not with discarded B. In the order B, A, C, it retains B and assigns both A and C to B. The chosen representatives therefore depend on input order. Structures assigned to the same representative need not match one another. To favor low-energy representatives, order the input Batch by energy before building an index; the returned indices then refer to that order. For a larger pool, process chunks in input order: combine previously retained representatives with the next chunk, deduplicate that combined Batch, and carry its representatives forward. Preserve representative order and carry original pool indices alongside each combined Batch; each result’s indices refer to that combined Batch. Use the same threshold and confirmation rule at every step. This limits the resident index to retained structures plus one chunk, at the cost of rebuilding it each time. If nearly every structure is unique, the retained set can still grow large. Deduplicating chunks independently and merging their survivors can change the result because similarity need not be transitive. Build new indices after geometry or atom labels change. See the RadialComparisonIndex reference for score dtype and CUDA memory settings.

For a single in-memory pool, use deduplicate_batch(). It tiles descriptor construction by total atom count and uses the same greedy representative order as RadialComparisonIndex.deduplicate. Set atom_types only when generic caller-defined labels should constrain matching; atomic numbers are never inferred as types.

For a pool loaded in pieces, use deduplicate_stream(). Its loader receives CPU int64 logical row IDs and returns (Batch, atom_types). The int32 or int64 type vector aligns with the atoms in that Batch, or is None on every call for untyped matching. Typed loaders must use one fixed type_vocabulary whose labels keep the same meaning in every batch. Loader values must remain stable for repeated row IDs. priority_order is a permutation of logical row IDs; when omitted, logical order is used. The first retained representative that passes every requested screen in that order receives each candidate, so changing the priority can change the result.

from nvalchemi.csp.comparison import deduplicate_stream

result = deduplicate_stream(
    count,
    read_typed_batch,  # row IDs -> (Batch, aligned atom_types or None)
    type_vocabulary=type_vocabulary,
    cutoff=15.0,
    threshold=0.05,
    priority_order=priority_order,
    device="cuda",
    max_batch_atoms=200_000,
    max_memory_fraction=0.85,
    summary_coordinate_count=32,  # default; use 0 to disable the preliminary check
)

When types are supplied, every proposed pair passes untyped, center-typed, and center-and-neighbor-typed screens in that order. An untyped loader uses only the untyped screen. In both cases, each same pair advances through the screens before the candidate is assigned. The result contains retained original row IDs in priority order, representative_indices indexed by original logical ID, and multiplicities aligned with retained IDs.

Pass confirm to deduplicate_batch or deduplicate_stream when a stronger comparison must accept a radial proposal before a structure is discarded. The callback receives (candidate_id, retained_representative_id) pairs and returns an ordered subset on the same device. Rejecting a proposal makes the candidate try later retained representatives; if none is confirmed, it is kept. Confirmation runs only against representatives already retained by the greedy pass. Match enumeration remains separate: callers can consume and filter chunks from iter_matches_stream without discarding any structures.

Here, confirm_pairs is your stronger comparison function; it returns the accepted input pairs in their original order. For an in-memory pool:

from nvalchemi.csp.comparison import deduplicate_batch

result = deduplicate_batch(
    batch,
    atom_types=topology_types,
    cutoff=cutoff,
    threshold=threshold,
    priority_order=priority_order,
    confirm=confirm_pairs,
)
deduplicated_batch = batch[result.retained_indices]

input_batch_size and pair_block_size default to 1024 and 256 on CUDA, and 64 and 32 on CPU. input_batch_size bounds candidate chunks; pair_block_size bounds how many earlier representatives are considered together. Candidate tiles can therefore be larger than representative blocks, subject to max_batch_atoms and the shared CUDA memory budget. max_batch_atoms defaults to 200,000 total atoms in one descriptor tile. A single structure larger than this limit raises MemoryError. CUDA descriptor construction and scoring budget their estimated buffers and workspaces to 85% of the memory currently available to PyTorch by default, including reusable caching-allocator bytes. This is an estimate, not a memory reservation. A single structure or pair that still cannot fit raises MemoryError.

By default, streaming deduplication uses a preliminary check selected by a deterministic pilot. The pilot selects up to 32 summary coordinates; it may select fewer. A pair that fails this conservative check is skipped. A pair that passes enters the radial index matcher, which may use its own conservative bounds before scoring full descriptors. The pilot summary never declares a duplicate on its own. Set summary_coordinate_count=0 to disable the pilot and its preliminary check; the index matcher’s built-in bounds still apply. The same option is available on deduplicate_batch and iter_matches_stream.

The summaries and sorted search columns are call-local and grow linearly with pool size and selected coordinate count. Compact summaries and sorted search columns reside on the comparison device. Full descriptors exist only for active atom-bounded tiles. Call state is released on return; the function creates no disk checkpoint or resume state.

To enumerate matches without supplying a pair list, use iter_matches_stream(). It accepts one loader for self-comparison or two loaders for a cross-pool comparison. The iterator starts reading on the first next() call, then yields nonempty int32 [K, 2] chunks no larger than pair_chunk_size. Self-comparison emits each left < right pair once; cross-pool comparison emits (left_row, right_row). Both orders are lexicographic. Closing the iterator releases its compact call-local state and leaves both caller-owned loaders open. max_batch_atoms bounds each live left or right descriptor tile independently; their combined CUDA descriptor residency remains within one shared memory allowance.

Compare without hydrogens#

Hydrogens can be omitted from comparison by passing include_hydrogens=False when building an index or calling a batch or streaming comparison helper. The library removes every atom with atomic number 1 from the internal comparison geometry before constructing neighbor lists and descriptors. Both centers and neighbors exclude hydrogens. The input structures remain unchanged.

Supply atom_types for the original, unfiltered atoms. Retained atoms keep their supplied types; chemical typing is not recomputed after removing hydrogens. Returned indices and confirmation-callback pairs refer to the original structure rows, so selecting retained rows preserves their hydrogens and other data.

An index fixes its hydrogen setting at construction. Indexes compared with each other must use the same setting; rebuild an index to change it. In streamed cross-pool matching, include_hydrogens applies to both pools.

Hydrogen-free comparison follows the existing empty-structure behavior of each API. Index construction rejects a row with no remaining atoms. Streaming matching treats empty rows as matching each other and separate from nonempty rows. An empty pool still returns empty results.

All-atom and hydrogen-free scores compare different geometries; removing hydrogens does not guarantee a smaller score. The radial comparison remains approximate and does not establish crystal equivalence.

from nvalchemi.csp.comparison import (
    RadialComparisonIndex,
    deduplicate_batch,
    deduplicate_stream,
    iter_matches_stream,
)

index = RadialComparisonIndex.build(
    batch,
    cutoff=15.0,
    atom_types=topology_types,  # Labels for every original atom, including H.
    include_hydrogens=False,
)
scores = index.score_pairs(pair_indices)  # Pair IDs remain original Batch rows.

result = deduplicate_batch(
    batch,
    atom_types=topology_types,
    cutoff=15.0,
    threshold=0.05,
    include_hydrogens=False,
)
retained_with_hydrogens = batch[result.retained_indices]

stream_result = deduplicate_stream(
    count,
    read_typed_batch,
    type_vocabulary=type_vocabulary,
    cutoff=15.0,
    threshold=0.05,
    include_hydrogens=False,
)
match_chunks = iter_matches_stream(
    left_count,
    read_left_typed_batch,
    other_count=right_count,
    read_other_typed_batch=read_right_typed_batch,
    type_vocabulary=type_vocabulary,
    cutoff=15.0,
    threshold=0.05,
    include_hydrogens=False,
)

Compare with an experimental structure#

Here, batch contains relaxed OverlapReliefPacker structures from packing_input, and template_adjacency contains formula-unit bonds in packing_input.atomic_numbers order.

An experimental CIF supplies cell and coordinates, but molecular bonds still need to be established. Resolve disorder or partial occupancy, use the same hydrogen choices as the candidate pool, and check that ASE reads a complete unit cell. Also check that the proportions of molecular components match the formula unit: a 1:1 two-component formula may appear as 2:2 in a cell, but not 2:1. Install nvalchemi-toolkit[ase] for CIF loading. Toolkit does not provide connectivity_from_reference; implement it for your CIF preparation workflow. It must return two Torch tensors in ASE atom order: integral atomic numbers [N] and a symmetric boolean molecular-bond matrix [N, N] with a zero diagonal. Bonds across cell boundaries must be included; nearby atoms are not necessarily bonded.

Build the type map from the same ordered formula unit used for the search. It matches complete molecular bond graphs to the formula unit, assigning the same numeric type labels even when the CIF lists atoms in a different order. Unknown molecular components are rejected. Typing does not check component proportions or infer missing bonds. If you ran the preceding deduplication example, reuse its type_map and typed_index instead of rebuilding them. For this standalone snippet, replace the ... placeholders with the same formula-unit input, bond matrix, and relaxed Batch.

from ase.io import read

from nvalchemi.csp.chem import TopologicalAtomTypeMap
from nvalchemi.csp.comparison import RadialComparisonIndex
from nvalchemi.data import AtomicData, Batch

packing_input = ...  # The same formula-unit input used for the candidate pool.
template_adjacency = ...  # Its bonds in packing_input.atomic_numbers order.
batch = ...  # Toolkit Batch of optimized trial crystals with ASU source fields.

cutoff = 15.0  # angstroms
threshold = 0.05
type_map = TopologicalAtomTypeMap(
    packing_input.atomic_numbers, template_adjacency
)
typed_index = RadialComparisonIndex.build(
    batch, cutoff=cutoff, atom_types=type_map.for_batch(batch), typed_neighbors=True
)

reference_atoms = read("experimental.cif")
reference_numbers, reference_adjacency = connectivity_from_reference(
    reference_atoms
)
reference_types = type_map.for_connectivity(
    reference_numbers, reference_adjacency
)
reference_batch = Batch.from_data_list(
    [AtomicData.from_atoms(reference_atoms, device=typed_index.device)]
)
reference_index = RadialComparisonIndex.build(
    reference_batch,
    cutoff=cutoff,
    atom_types=reference_types,
    typed_neighbors=True,
)
experimental_matches = typed_index.find_matches(
    other=reference_index, threshold=threshold
)
experimental_scores = typed_index.score_pairs(
    experimental_matches, other=reference_index
)

Each row of experimental_matches is [candidate_index, 0]: column zero identifies a structure in the original search pool, and zero identifies the single experimental reference. experimental_scores has one mismatch score per returned pair. A low score proposes a match for more detailed structural comparison; it does not establish that the structures are crystallographically equivalent. For additional CIFs, repeat the reference preparation and matching block with the same type_map and typed_index.

Advanced data and storage details#

The formula-unit input owns its conformer pool, contact distances in \(\mathrm{\AA}\), component IDs, and volume estimate in \(\mathrm{\AA}^3\). An ASU batch shares that input and stores each candidate’s cell, space group, and independent placements. Cell vectors are rows of a 3 x 3 matrix in \(\mathrm{\AA}\). Fractional molecular centers are dimensionless and become Cartesian through fractional @ cell; rigid rotations act on Cartesian molecular displacements. Expansion emits A_fu * Z atoms per cell, where A_fu is the atom count of one formula unit. Molecular centers are wrapped; atom coordinates are not wrapped separately.

Tensor-only callers can construct MolecularPackingInput and RigidMoleculeASUBatch directly. Pointer arrays mark spans in the next array; [0, 2] means the first item occupies indices 0 and 1. See their API pages for shapes and validation rules. Direct callers must supply valid rotation matrices and cells compatible with the selected space groups.

Source metadata after geometry changes#

Full-cell expansion records where every atom and structure came from. These fields are provenance, even if subsequent optimization breaks source symmetry:

Field

Alignment

Meaning

csp_source_asu_atom_index

Atom

Atom in the repeated ASU order

csp_source_molecule_index

Atom

Molecule in the repeated ASU order

csp_source_component_index

Atom

Component from the formula unit

csp_source_conformer_index

Atom

Conformer in the shared input

csp_source_symmetry_operation_index

Atom

Operation that generated the copy

csp_source_space_group

System

Source International group number

csp_source_z

System

Source formula units in the cell

csp_source_z_prime

System

Source formula units in the ASU

csp_source_structure_id

System

int64 [run_id, accepted_structure_number]

For Z' > 1, ASU atom and molecule indices include every repeated independent formula unit. Toolkit Batch selection and cloning preserve aligned fields. To type a saved relaxed Batch later, retain the ordered formula-unit atomic numbers and bond matrix once alongside the search results and rebuild its TopologicalAtomTypeMap. Source indices alone do not identify the molecular graph. Keep pools from different formula-unit inputs separate when assigning or comparing type IDs.

Store maintenance and reproducibility#

read() and delete() address the current undeleted row order; deleting row 0 makes old row 1 the new row 0 without changing IDs. Existing readers must refresh() after writes. Use one active writer per store and reopen it if another writer changes the store. defragment() reclaims deleted-row space and needs exclusive access. If local-directory replacement is interrupted after moving the original, restore the sibling .NAME.csp-backup-* directory before reopening. Custom property names cannot contain / or be ., .., or zarr.json.

Without an explicit run ID, a direct OverlapReliefPacker call or CSPGenerator generates one independently of the Torch RNG. Distributed IDs are [run_id, group_rank + group_size * local_accepted_number]; gathered rows follow rank order. For replay, retain the run ID, each rank’s seed and settings, formula input, group size, and rank mapping. Changing group membership can change candidates. Startup validation is coordinated on live ranks; packing errors are local in rank-local mode and exchanged at terminal gathering. Process termination cannot be coordinated by CSPGenerator.