Distilling a Teacher Into a Student#

Knowledge distillation trains a small model — the student — to reproduce the predictions of a larger, frozen one — the teacher. For interatomic potentials the motivation is throughput. A foundation teacher may be far too expensive to drive long molecular dynamics, while a student that reproduces its energies and forces on the states that matter runs orders of magnitude faster. The teacher also removes the usual data bottleneck, because it can label any structure, including ones no reference calculation was ever run on.

DistillationStrategy is the entry point. It is a TrainingStrategy subclass, so everything in Training — optimizers, schedulers, validation, hooks, checkpoints — applies unchanged. This guide covers what distillation adds: the offline path, which labels a dataset with the teacher once, and the on-policy path, which labels the structures the student visits in its own dynamics. The symbols are in Distillation API.

This guide assumes that you already have:

  • a teacher wrapped with BaseModelMixin;

  • a student that is trainable and declares the outputs your objective reads;

  • a dataset of structures, with or without reference labels.

For those prerequisites, see Models: Wrapping ML Interatomic Potentials, Data Loading Pipeline, and Training.

The shape of a distillation run#

DistillationStrategy takes a named-model mapping holding "student" and "teacher". The teacher is frozen by omission: it must not appear in optimizer_configs. That absence puts it in evaluation mode, with gradients disabled, for the duration of the run. The student, and any auxiliary model such as a learned projection, must be given an optimizer config. The strategy raises at construction if either half of that contract is broken.

Teacher knowledge reaches the loss as ordinary batch fields. A teacher signal is one quantity the teacher labels a batch with, such as its energy or its forces. Each signal populates one teacher_* field, and a loss term consumes it by pointing its target_key there:

from nvalchemi.training import EnergyMSELoss, ForceMSELoss
from nvalchemi.training.distillation import AtomicEnergyMatchingLoss

loss_fn = (
    EnergyMSELoss(target_key="teacher_energy")
    + ForceMSELoss(target_key="teacher_forces", normalize_by_atom_count=True)
    + 0.2 * AtomicEnergyMatchingLoss()
)

Distillation therefore needs no special loss machinery: any built-in term distills by naming a teacher field. Offline, where every sample carries its own reference labels, mixing teacher targets with reference targets in one objective is ordinary loss composition. An on-policy run cannot mix them, for reasons covered below. The built-in signals, and the field and shape each one lands as, are:

Signal

Batch field

Level

Shape

energy

teacher_energy

system

(B, 1)

forces

teacher_forces

node

(V, 3)

stress

teacher_stress

system

(B, 3, 3)

atomic_energies

teacher_atomic_energies

node

(V,)

embeddings

teacher_node_embeddings

node

(V, D)

hessian

teacher_hvp, with teacher_hvp_probe

node

(V, 3)

The first four come from the teacher’s forward pass. Each shares its name with the output the teacher must declare in ModelConfig.outputs, and that name is what the construction check reports as missing. teacher_node_embeddings comes from compute_embeddings(), which costs a second pass. teacher_hvp comes from a Hessian-vector product along a random probe direction, which the scorer stores beside it in teacher_hvp_probe.

Each row is a TeacherSignal, and the table is open. To label any other teacher output, pass a spec of your own beside the built-in names. TeacherSignal("charges", "charges", "teacher_charges", "node") reads the teacher’s charges output into a node-level teacher_charges field; an optional normalize= callable gives the output the shape the field should land as. Three rules are checked at construction: the field must start with teacher_, the level must be node or system, and the scorer refuses a spec naming an output the teacher does not declare. The built-in specs are published as BUILTIN_SIGNALS.

embeddings and hessian are the two signals the stock training function cannot supervise on its own, because neither has a student-side counterpart in a plain forward pass. For those, use embedding_distillation_fn() or hessian_distillation_fn() (see Objectives beyond pointwise matching). Using the default training function with embedding prediction keys raises a construction error directing you to these functions.

You do not normally declare which signals you want. With teacher_signals=None, the default, the strategy derives the set from the teacher_* targets the losses read: the training loss, plus the validation loss when validation_config carries a loss_fn of its own. The objective and the teacher therefore cannot drift apart. An explicit set must cover the derived one and may request more. At construction, the strategy checks that the teacher declares every resolved signal in outputs. It also checks that the student computes every loss prediction key, reading active_outputs intersected with the declared outputs, so a pretrained wrapper whose active set was narrowed fails at construction rather than on its first batch.

These checks do not run on property assignment. Pass validation_config to the constructor, or declare the extra signals in teacher_signals, rather than assigning either afterwards. A spec leaves validation_config out, because it carries a live loader, so a rebuild takes it as a runtime override alongside the models: from_spec_dict(spec, models=..., validation_config=...), and the same keyword on from_checkpoint_dict and load_checkpoint. A restored run that validates against a teacher_* target its training loss does not read needs validation_config passed there rather than assigned afterwards.

Every resolved signal is a request for its fields on every batch. A batch counts as labeled only when it holds every resolved field. Suppose a validation loss reads a teacher_* target that the training loss does not: a store labeled without that field then sends every batch back to the teacher. A store meant to train with no teacher pass at all has to be labeled with the same signal set the strategy resolves. A teacher_* target that names no built-in signal, such as a field a custom scorer persisted, is an ordinary loss target. It is read off the batch as it arrives, never derived into a signal, and never attached on the fly, so a batch lacking it surfaces as a missing loss target.

The training function stays a plain student forward pass. The default, default_distillation_fn(), calls the student and prefixes every output with predicted_. The teacher is never called there. That is why the teacher can never enter the student’s autograd graph, and why a spec can name the training function by its import path.

AtomicEnergyMatchingLoss is the one pointwise term distillation adds. It matches the teacher’s per-atom energy decomposition, a quantity no reference-labeled dataset carries. Per-atom energies are not physically observable on their own, so treat the term as a regularizer on the student’s internal decomposition, and keep a total-energy term weighted above it. The term asks the most of both models, because the decomposition has to exist on each side. The teacher must declare atomic_energies in ModelConfig.outputs for the signal to be scorable. The student must both declare and compute it, since the term reads prediction_key="predicted_atomic_energies". A student that emits only energy and forces, including DemoModelWrapper, therefore fails at construction on the three-term objective above, with an error naming the loss component and the missing atomic_energies.

Objectives beyond pointwise matching#

Three further terms distill what a pointwise target cannot carry, and each asks the run for something a plain forward pass does not produce. The weighting and the physics behind each are in Embedding, Hessian, and Boltzmann objectives. This section covers what the run has to provide.

EmbeddingMatchingLoss matches the teacher’s per-atom representation rather than a prediction. Both sides come from compute_embeddings, so the objective needs embedding_distillation_fn() as the training_fn, and the student runs twice per batch. Widths rarely agree across architectures, so register an EmbeddingProjector under the model name "projector" and give it an optimizer config. The training function routes the student’s embeddings through it. At construction, the strategy checks that the student’s width, the projector’s in_features/out_features, and the teacher’s width compose. It refuses a student that publishes no node_embeddings shape. It also refuses a student whose compute_embeddings returns embeddings detached from its trainable parameters, because the projector would then absorb the whole objective while the student learned nothing from it. When that is the intent — a trunk frozen on purpose beside a trainable head, with the projector alone carrying the term — register the projector with frozen_student=True. The strategy refuses that flag when every student parameter is trainable.

BoltzmannMatchingLoss is the one term that scores a batch rather than a sample. It matches the Boltzmann weights that the two energy surfaces imply over the batch at a temperature, in Kelvin. beta is not an inverse temperature: it interpolates between the forward (0) and the reverse (1) relative entropy. The forward direction is bounded by log B across B scorable graphs. Its gradient vanishes when the softmax saturates, which happens once the student’s energy errors exceed a few k_B T. Under beta=0, a saturated gradient can look like convergence while the student’s errors remain large, so keep beta >= 0.5 until the errors drop within a few k_B T.

The Boltzmann term reads a batch as a sample of the student’s own Boltzmann distribution. It therefore requires on_policy, and it refuses a relaxation propagator and any convergence criterion. It warns when replay_ratio mixes in samples the student never visited; the recommended shape is replay_ratio=1 with a bounded replay_capacity. It is refused on the validation side, because a fixed holdout is not a sample of the student’s distribution. That covers a validation loss_fn that holds the term, and a validation config with no loss_fn of its own that would reuse a training loss holding it. The refusal of a relaxation propagator or a convergence criterion rests on whether the propagator samples an equilibrium ensemble at all. The strategy infers that unless OnPolicySettings.samples_equilibrium says otherwise. That setting is the override for a propagator the inference misreads, and the refusal names it.

Under a DDPHook the softmax runs over the world batch. The reduced energies are gathered across ranks differentiably, so every rank reports the world loss. The one-system check reads that gathered batch, so a rank holding one system beside a rank holding a system of a different size is refused too. The check compares atom counts alone: it refuses a batch whose graphs differ in size, and it cannot tell two systems of the same size apart. The gather and the check are constructor settings. world_batch=None, the default, gathers only when a process group with more than one rank is initialized, and True or False forces it. check_one_system=False drops the guard for a batch you know to be comparable. With ignore_nonfinite (the default), a graph whose teacher or student energy is not finite is dropped from the ensemble rather than reaching every rank’s softmax. A batch of one graph scores exactly 0.0.

HessianMatchingLoss matches the teacher’s curvature along a random probe direction. The student’s product is taken by hessian_distillation_fn() on a second, energy-only pass that reuses the neighbor list the stock forward just ran on. Under the default normalize_by_atom_count=True the term is graph-balanced: every graph weighs the same whatever its atom count. The reduction is the core graph_balanced_mean(), which the atomic-energy, embedding, and Hessian terms share and a term of your own can call; normalize_by_atom_count=False takes one global mean over the valid components instead. The graph-balanced value carries the square of a force constant’s units. For a near-converged student it runs one to two orders of magnitude above a force mean-squared error, so start the term a hundred to ten thousand times lighter than the force term. Read one batch’s value as the one-sample estimate it is. Set the weight on the composition rather than on the term, since leaves are weightless and ComposedLossFunction renormalizes by default. A loss pointed at the companion teacher_hvp_probe is refused, since a probe is not a quantity the student is supervised against. A student whose forces come from a head of their own is warned that the term supervises its energy head alone.

The Boltzmann term also changes what a restart needs. It is defined on generated batches, so it refuses to be rebuilt without the on-policy loop described below. When the checkpoint’s spec could not carry that loop whole, re-supply it at load time, as load_checkpoint(root, models=..., on_policy=..., reference_dataset=...), or restore into a strategy already built with them. The Hessian term has no such requirement: it reads only what the student computes on the batch in hand.

Offline distillation#

The offline path scores the dataset once and trains from the result. It is the cheaper path by a wide margin whenever the same structures are visited more than once, and it is where any distillation project should start.

Label the dataset once#

label_dataset() walks a dataset in chunks, scores each chunk with a TeacherScorer, and writes the source fields plus the teacher fields into a Zarr store:

from nvalchemi.training.distillation import InProcessTeacherScorer, label_dataset

scorer = InProcessTeacherScorer(teacher, ["energy", "forces", "atomic_energies"])
num_labeled = label_dataset(dataset, scorer, "labeled.zarr", batch_size=64)

InProcessTeacherScorer handles the teacher’s evaluation contract for the caller. It narrows the teacher’s active_outputs to what the requested signals need, builds the teacher’s own neighbor list and rolls it back, picks the grad mode a teacher with autograd outputs requires, detaches every tensor it returns, and normalizes each signal to the canonical shape above. It leaves the batch it is handed exactly as it found it, so the same scorer works mid-training and mid-trajectory. dtype= stores labels at a reduced dtype. probe_seed= pins the Hessian probe direction; leave it unset for training and labeling, where coverage comes from redrawing the probe.

The neighbor_list= setting decides where the teacher’s neighbor list comes from. The default, "rebuild", builds the teacher’s own list on every call, whatever the batch carries. "reuse" is for a student that has already built the list the teacher needs, in the teacher’s format and at its cutoff. The scorer then consumes that list and builds nothing. It checks only what it cannot infer: that the keys the teacher’s format reads are present, and that a cutoff stamp, if the batch has one, equals the teacher’s cutoff. Otherwise it raises a ValueError naming the missing key or the mismatched cutoff; it never falls back to a rebuild. The batch does not record whether a list holds each pair once or twice, so a reused list has to match the teacher’s half_list by construction.

The scorer builds exactly one list per batch, so it refuses a PipelineModelWrapper teacher that plans more than one neighbor-list source; compose such a teacher with neighbor_adaptation="always" or a wide enough max_cutoff_ratio (see Models: Wrapping ML Interatomic Potentials).

Labels are attached with overwrite=True. A scorer reaching outside the teacher_* namespace would therefore replace the reference field of that name, which is the very label the student is trained against. label_dataset refuses declared label_fields outside the namespace before it writes the first chunk, and it checks the fields each chunk actually returns again. The same namespace rule holds on every other labeling route.

Labeling is resumable. An existing store is continued from len(store), and every resumed chunk is checked against the store’s fields, levels, dtypes, and row shapes. Two stores are refused rather than resumed from a misaligned offset: one longer than the dataset, which was written from a different dataset, and one that an interrupted append left inconsistent. The neighbor list is not carried over, in either format. The dense tensors cannot append into a fixed-width store array, and a sparse list is dropped because the cutoff it was built at is a batch attribute the store does not hold. keep_neighbors=True stores the sparse list anyway. A store written under one setting and resumed under the other is refused as field-set drift. Nothing rebuilds a list when batches are read back out of the store; the next section adds the hook that does.

Train from the labeled store#

Reading the labeled store back is not distillation-specific. The teacher fields arrive as ordinary batch attributes at the levels they were written at. The reader, dataset, and loader are the ones any training run uses, and no teacher forward pass happens during training at all:

import torch

from nvalchemi.data.datapipes import AtomicDataZarrReader, DataLoader, Dataset
from nvalchemi.training import OptimizerConfig
from nvalchemi.training.distillation import DistillationStrategy

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
loader = DataLoader(
    Dataset(reader=AtomicDataZarrReader("labeled.zarr"), device=device),
    batch_size=32,
)

strategy = DistillationStrategy(
    models={"student": student, "teacher": teacher},
    optimizer_configs={
        "student": [OptimizerConfig(optimizer_cls=torch.optim.Adam)]
    },
    loss_fn=loss_fn,
    num_steps=10_000,
    devices=[device],
)
strategy.run(loader)

Open the store on the device the strategy trains on, and name that device on both sides: a Dataset opened without device= emits on the current CUDA device whenever one exists, while devices defaults to CPU and takes torch.device objects (see Training strategy API). label_dataset takes the same device= and moves each chunk there before scoring; put the teacher there yourself with teacher.to(device).

Nothing in the training loop builds a neighbor list. Labeling drops the neighbor tensors from the store, and a wrapped MLIP raises rather than building its own list, so a graph student needs a NeighborListHook at TrainingStage.BEFORE_FORWARD in hooks=, configured from its own neighbor config. Graph students need a neighbor list on both engines below shows it beside the propagator-side hook the on-policy loop needs as well, and AtomicData and Batch covers the list formats. Both examples use neighbor-free demo potentials, so the hook becomes necessary only when a real MLIP takes the student’s place.

A complete, runnable version of this workflow is Offline Knowledge Distillation.

Consuming the labeled store elsewhere#

The store is a plain Zarr hierarchy, so the labels are not locked to this toolkit. teacher_energy and teacher_forces are arrays under the core group beside positions, with the per-sample offsets of per-atom arrays in meta/atoms_ptr (the layout is in Zarr Compression Tuning), and the store’s root attributes record whether each field is per-atom or per-system. Any Zarr client can therefore read a labeled store: a student written in another framework consumes the same teacher labels without running the teacher again, and without importing nvalchemi at all.

Labeling on the fly#

A batch that arrives without the required teacher_* fields is labeled on the fly instead. The strategy registers an internal hook for this on BEFORE_FORWARD, prepended ahead of your own hooks, so labeling happens before a neighbor-list hook of yours runs; that order costs nothing, because the scorer builds and rolls back the teacher’s own list whatever is on the batch. Short runs and interactive sessions therefore work with no labeling pass at all, and unlabeled validation data needs no preparation. The first batch labeled this way triggers a one-time warning that names the teacher fields it lacked, so a long run does not pay an unplanned teacher pass per step unnoticed. Set label_missing=False to turn on-the-fly labeling off; an unlabeled batch then surfaces as a missing loss target.

On-the-fly labels are attached to the device-placed batch the strategy trains on. That batch is a copy of the one you handed over, so the labels do not persist on your object. A loader that replays the same systems every epoch therefore pays one teacher pass per epoch, which is why a long run should label offline first. Label precision is the scorer’s: InProcessTeacherScorer(autocast=False) (the default) disables autocast around the scoring pass whatever region surrounds it, so an on-the-fly label equals the offline one and the two paths are interchangeable. autocast=None runs the teacher under the caller’s region; True or a floating dtype enables it (True keeps the ambient region’s dtype, a dtype pins the pass).

On-policy distillation#

Offline distillation trains the student on whatever structures the dataset happens to hold. Those are not the structures the student visits once it drives dynamics itself, and that gap is what makes a distilled potential drift or blow up on long trajectories. On-policy distillation closes the gap: the student’s own propagator generates frames, the teacher labels them, and the student trains on them.

Setting on_policy turns run() into a segment loop. A segment is one generate-label-train cycle, and the loop repeats segments until the step budget is spent. run() then takes no dataloader, because each segment builds its own. One segment has three phases:

  1. Generate — the propagator advances the live batch by generation_steps. The first segment starts from the initial structures, the structures the trajectories begin at, passed as initial_structures.

  2. Label and capture — the labeling hook, a TeacherLabelHook on the propagator, scores a frame every label_frequency steps. It copies each labeled frame into the capture sink, the DataSink that holds a segment’s frames until the segment boundary. The capture sink is host memory unless capture_sink names another sink; a GPUBuffer keeps the frames on the device. The segment’s final frame is labeled too. The sink is then drained into the replay buffer, a ReplayBuffer that accumulates labeled frames across segments.

  3. Train — a freshly built mixed loader draws training_steps_per_segment batches, and each batch goes through the ordinary per-batch stages. Every batch is a mixture of replay-buffer frames and samples from the reference dataset, a fixed teacher-labeled dataset, split at replay_ratio (see The mixture ratio).

from nvalchemi.dynamics.integrators.nvt_langevin import NVTLangevin
from nvalchemi.training.distillation import InitialStructures, OnPolicyConfig

strategy = DistillationStrategy(
    models={"student": student, "teacher": teacher},
    optimizer_configs={
        "student": [OptimizerConfig(optimizer_cls=torch.optim.Adam)]
    },
    loss_fn=loss_fn,
    num_steps=10_000,
    on_policy=OnPolicyConfig(
        dynamics=NVTLangevin(student, dt=0.5, temperature=300.0, friction=0.01),
        teacher_scorer=scorer,
        initial_structures=InitialStructures(initial_dataset),
        replay_ratio=0.25,
        training_steps_per_segment=32,
        batch_size=16,
        generation_steps=50,
        label_frequency=10,
        replay_capacity=8192,
    ),
    reference_dataset=reference_dataset,
)
strategy.run()

The propagator must hold the very module registered as models["student"], either directly or composed into a larger model. The strategy checks that object identity at construction. Because the module is shared, every optimizer update reaches the trajectories the propagator generates next. A propagator that merely composes the student is moved whole to the generation device and held in evaluation mode for the whole loop, because the training phase forwards models["student"] rather than the composition. The mode contexts are the core evaluating() and eval_configured_models() (see Training strategy API).

The run is sized in optimizer steps rather than epochs, since each segment builds its own loader. One segment counts as one epoch for hooks and epoch-cadence validation, while a step-cadence validation fires inside segments. The run closes with one terminal validation, skipped when a cadence already validated at the final step, so a metric-driven scheduler is never stepped twice on one set of metrics.

The loop is data-parallel across ranks. Each rank propagates its own strided shard of the initial structures, labels those frames with its own teacher replica, and fills its own replay buffer. The student’s gradient all-reduce, run by a DDPHook, is the only per-step training traffic between ranks. Scaling the segment loop out below covers what a multi-rank run asks of the settings.

The propagator is any BaseDynamics — an integrator generating trajectories, or an optimizer generating relaxation paths. Nothing downstream of the config reads a velocity or a temperature.

The initial structures have to carry the fields the propagator updates in place. Building the OnPolicyConfig loads one row from initial_structures and hands it to the propagator’s check_initial_batch(), so a missing field is a construction error named against the propagator rather than an AttributeError on the first step (Implementing Custom Dynamics covers which keys that method reads). For the built-in propagators the check comes to velocities and atomic_masses, which AtomicData fills unless a store they were written to dropped them. The variable-cell propagators also need a cell, because nothing fills one in for an aperiodic structure.

You do not need to supply the keys listed in __needs_keys__: forces for every integrator and optimizer, plus stress for NPT, NPH, and the variable-cell FIRE2 and L-BFGS optimizers. The propagator computes them with one compute before its first step. OnPolicySettings.probe gates that construction-time forward, and the criterion probe a relaxation lifecycle runs on the same row; probe=False skips both and defers a mismatch to the first step. The probe builds exactly one neighbor list, so it cannot run a propagator whose model plans more than one neighbor-list source. Such a propagator is named in a warning, which probe=False also silences.

Initial structures therefore need not differ from reference-dataset samples, which must carry no energy or forces at all. They may also be the outputs of earlier relaxation runs. Such a store marks its structures with an exit status, which the propagator would otherwise read as finished and refuse to move. InitialStructures strips that metadata and re-initializes status and system_id, so the propagator can advance the structures without a manual cleanup pass.

The initial structures live behind an InitialStructures: a thin subclass of the core OrderedStructureSampler that adds only the to_spec_dict/from_spec_dict round trip. The sampler serves the dataset’s rows in order from one position, next_row, which the initial batch, any later backfill (the structures a relaxation lifecycle draws to replace the ones that finished) share. A structure is therefore propagated once per pass over the rows. The position is not part of the checkpoint: the segment loop re-shards the sampler at every start, which resets it, so a resumed run reseeds from the front of its shard (see Operational notes) and may generate from structures the interrupted run already visited; Structure sources covers the position, the budgets, draw, and sharding. initial_structures= takes any StructureSource, the protocol of the members the loop reads, importable from the distillation package under its historical name, InitialStructuresSource. A bare BatchDatasetProtocol dataset, the common case, is wrapped in an unbudgeted source. See Custom components for a source of your own.

An unbudgeted source is propagated whole, as a single batch, so it is the set of systems the run generates from; size it to the device. A budget — InitialStructures(dataset, max_atoms=4096), or max_batch_size, or max_edges — packs the initial batch first-fit in row order instead and leaves the remainder, in row order, for the backfill (see Relaxation paths need a convergence lifecycle).

label_frequency is the throughput setting. The teacher is the expensive model. A segment that labels every tenth frame costs a tenth of the teacher passes, while it still generates every frame at student speed. The cadence counts the propagator’s cumulative step_count, which carries across segments. The hook reads the count before it is incremented, so a frequency f fires at steps 0, f, 2f, ..., while a segment’s forced last frame lands one step later. When generation_steps is a multiple of label_frequency, as with the defaults (100 and 100), the cadence and boundary labels would land on adjacent steps. To avoid scoring twice when label_frequency > 1, the hook skips the cadence dispatch that immediately follows a labeled frame. Forced boundary labels are always kept, so an early-exiting segment and the final frame of the run are still labeled. Under the defaults that leaves one label per trajectory per segment, on its last frame. The very first segment adds one more: the frame after the first step, which the cadence lands on at step count zero. step_count never resets, so a later run() or a restart does not pay that extra label again.

Size replay_capacity with that arithmetic in hand. A segment contributes one frame per trajectory per labeled step, and FIFO eviction retires whole frames in arrival order. A capacity that is not a multiple of the trajectory count therefore cuts a segment’s contribution mid-step. The trajectories at the back of the batch are then over-represented in every mixture drawn afterwards, so make the capacity a multiple of the number of initial structures. Two decisions are left to policy: what enters the buffer and what leaves it. replay_admission takes an AdmissionPolicy, a predicate over each segment’s frames applied before the schema check. replay_eviction takes the string "fifo", which means the FIFO reference and is the one spelling OnPolicySettings.replay_eviction carries, or an EvictionPolicy instance that names the frames a full buffer drops.

Note

Request the same signals on OnPolicyConfig.teacher_scorer that the loss reads. With a reference dataset there is no choice. The generation scorer’s teacher fields and the reference dataset’s stored ones are compared for equality at construction, so a scorer narrower than the dataset is rejected outright. A narrow scorer constructs only against an equally narrow dataset, and then every generated frame is scored twice: once during generation, and again on its way into a training step. The strategy warns whenever the generation scorer is narrower than the loss, with or without a reference dataset.

Any TeacherScorer may drive generation, not only the in-process one. A custom scorer should declare label_fields, the batch fields its label() populates. The strategy reads that declaration through scorer_fields() to check the targets up front. Without label_fields, the strategy cannot inspect the targets before scoring. It then warns and defers the parity check to the first segment’s mixed loader, so a mismatch surfaces only after a whole generation phase has been paid for. A label_fields entry outside the teacher_* namespace is refused at construction, because the labeling hook must never overwrite the energy and forces that drive the propagator’s next step. A custom teacher_* field the scorer writes is an ordinary loss target, as it is offline. Generation writes it onto every captured frame. The reference dataset has to carry it too, which the parity check enforces. Validation data has to arrive already carrying it, since nothing labels it on the fly. At least one built-in teacher_* target, or an explicit teacher_signals, is still required alongside it.

A runnable three-segment loop is On-Policy Knowledge Distillation.

Graph students need a neighbor list on both engines#

Both examples generate with neighbor-free demo potentials. The configuration above fails for a student that reads a neighbor list off the batch, for three reasons. The initial batch comes from a store, and a store holds no neighbor tensors. The construction check reads the propagator’s __needs_keys__, which never includes them. And nothing in a dynamics step builds a list. A wrapped MLIP therefore raises a KeyError naming the missing neighbor tensor on the first propagator step, before a single frame is generated. The training side has the same gap. The labeling hook strips the neighbor tensors from every captured frame, and label_dataset drops them from the reference dataset, so the first mixed batch also reaches the student’s forward without a list.

The remedy is one hook per engine, both configured from the student’s own neighbor config. The propagator takes a NeighborListHook at BEFORE_COMPUTE, and the strategy takes one at BEFORE_FORWARD:

from nvalchemi.dynamics import DynamicsStage
from nvalchemi.dynamics.integrators.nvt_langevin import NVTLangevin
from nvalchemi.hooks import NeighborListHook
from nvalchemi.training import OptimizerConfig, TrainingStage
from nvalchemi.training.distillation import (
    DistillationStrategy,
    InitialStructures,
    OnPolicyConfig,
)

neighbor_config = student.model_config.neighbor_config
propagator = NVTLangevin(student, dt=0.5, temperature=300.0, friction=0.01)
propagator.register_hook(
    NeighborListHook(neighbor_config, stage=DynamicsStage.BEFORE_COMPUTE)
)
strategy = DistillationStrategy(
    models={"student": student, "teacher": teacher},
    optimizer_configs={
        "student": [OptimizerConfig(optimizer_cls=torch.optim.Adam)]
    },
    loss_fn=loss_fn,
    num_steps=10_000,
    hooks=[NeighborListHook(neighbor_config, stage=TrainingStage.BEFORE_FORWARD)],
    on_policy=OnPolicyConfig(
        dynamics=propagator,
        teacher_scorer=scorer,
        initial_structures=InitialStructures(initial_dataset),
        replay_ratio=0.25,
        training_steps_per_segment=32,
    ),
    reference_dataset=reference_dataset,
)

student.make_neighbor_hooks() builds the propagator-side hook from the same config (see Models: Wrapping ML Interatomic Potentials). The teacher needs neither hook: the scorer builds and rolls back its own list on every batch it labels, so the student’s neighborhoods never reach it. The propagator’s hook is a live object that no spec describes. A loop rebuilt from a checkpoint therefore starts without it, and fails the same loud way on its first step. Register the hook again on strategy.on_policy.dynamics before run(), or restore into a strategy built with it. A student whose forces are an energy gradient also has to take them with create_graph=True while training, as DemoModelWrapper does; that is a wrapping matter covered in Models: Wrapping ML Interatomic Potentials.

Relaxation paths need a convergence lifecycle#

A relaxation propagator differs from an integrator in one way that matters here: its trajectories end. Without convergence tracking, the propagator keeps stepping converged structures and the labeling hook keeps copying them. The replay buffer then fills with redundant minimum configurations every label_frequency steps, while the run reports plausible losses. Set fmax to turn on the convergence lifecycle, which tracks convergence and graduates converged structures, retiring them from the batch:

from nvalchemi.dynamics import FIRE2
from nvalchemi.training.distillation import InitialStructures

on_policy = OnPolicyConfig(
    dynamics=FIRE2(student, dt=0.05),
    teacher_scorer=scorer,
    initial_structures=InitialStructures(initial_dataset, recycle=True),
    fmax=0.05,
    replay_ratio=0.25,
    training_steps_per_segment=32,
    batch_size=16,
    generation_steps=50,
    label_frequency=10,
)

fmax is the max-force-norm threshold. It is compared against the student’s forces, the ones the relaxation itself converges on. A criterion the threshold cannot express is passed whole, as convergence_hook=, instead. The two are one criterion under two spellings, so setting both is refused. A hook passed whole has to migrate status from the 0 the run stamps its structures with to at least the propagator’s exit_status (status migration is covered in Hooks, and graduation in Dynamics: Optimization and Molecular Dynamics), and it has to fire on every step, because a structure is captured at the step it converges. Prefer the threshold unless the criterion genuinely needs a hook. convergence_hook is a runtime-only field: it holds a live object that no spec can describe, so a rebuilt run needs it supplied again, while fmax is a plain number OnPolicySettings holds. Either way, OnPolicyConfig.convergence_criterion is the live hook the lifecycle drives, built once and handed over by identity.

The lifecycle alone manages graduation and backfilling. To prevent conflicting status transitions, a propagator that already holds a status-migrating ConvergenceHook, a multi-sub-stage FusedStage (constructing one registers a migrator on every non-last sub-stage), and a DomainParallel propagator (its step dispatches no ON_GRADUATE) are refused when the OnPolicyConfig is built. A migrator registered afterwards, or an internal sampler on the propagator, is refused when run() starts. The only fused shape the lifecycle accepts is a single sub-stage with no criterion of its own. Relaxation paths that genuinely need staged dynamics should use the propagator’s own lifecycle instead, with fmax left unset. A propagator managing its own convergence keeps its final frames, since graduated frames are left out of capture only under the managed lifecycle.

With fmax set, a converged structure freezes where it stopped and graduates on the step its status reaches exit_status. The propagator reports that step at DynamicsStage.ON_GRADUATE with ctx.graduated_mask, where a converged-frame hook stores the structure once, unlabeled, as the minimum it reached; the segment boundary retires it from the active batch and backfills its place. Frames then reach the buffer by two routes that partition them. The labeling hook stores the structures still relaxing, narrowing the frame to them before the teacher pass, so the teacher pass shrinks as the batch converges. The converged structures are labeled in a single teacher pass as their sink drains onto the buffer’s device.

A trajectory can also end by diverging. No criterion accepts a NaN, so a structure whose positions or forces stop being finite is frozen at exit_status on that step and kept out of both routes. It is retired and backfilled at the boundary like a converged one, and each boundary warns once with a count of such structures. OnPolicyConfig.divergence decides what counts as diverging: a callable over the frame batch that returns one boolean per graph. The default is nonfinite_divergence() (an alias of the core nonfinite_graph_mask(), over positions and forces). The predicate is evaluated once per step, and its verdicts are ORed into a record that the capture hook and the segment boundary both read. A predicate of your own, such as a bond-length or cell-volume bound, is a runtime-only field like replay_admission. A predicate returning anything but one bool per graph is refused.

The backfill draws from the same position the initial batch was packed from. It draws as many structures as graduated, within the atoms they held, and it skips a row that does not fit rather than stalling on it. An unbudgeted source has nothing left to draw, because it started every row it owns, so the batch narrows by one trajectory per graduation. InitialStructures(dataset, recycle=True) wraps the position back to the front instead, so the trajectory count holds and the run relaxes the same structures again. The structures are reloaded from the dataset as it stores them, not from where the propagator’s last frame left them, so the second pass starts from the same geometries under a fresher student. One draw reaches every row at most once, so two copies of one structure never enter a batch together. recycle is the source’s flag, and only a run managing a lifecycle ever backfills, so setting recycle with fmax unset is refused at construction. A budget is the other way to keep the batch full: the source packs a first-fit batch and spends the remaining rows on the backfill without serving a structure twice. When the position reaches the end with nothing left and no recycle, the run warns once and spends its remaining training steps on the frames it already has.

A thermostatted integrator has the opposite problem: its trajectories never end, so without a lifecycle the initial batch is the only set of structures the run ever propagates, and the teacher keeps labeling new geometries of the same molecules. max_trajectory_steps gives every trajectory a lifetime in propagator steps. A trajectory that reaches it is frozen on that step, stored once through the converged route, and retired and backfilled at the segment boundary exactly like a converged one, so the run rotates through the whole source. It turns the lifecycle on by itself and also works beside fmax, for a relaxation that should give up on a structure that never converges. A multiple of generation_steps retires on segment boundaries, so no frozen trajectory idles inside a segment; a source the run outlives needs recycle=True. Under a lifetime the backfill stops at the first row that does not fit the freed room instead of passing over it: every trajectory retires in turn, so the room grows to the full budget and that row is served then, and the pool is consumed in order with no row lost. A row larger than the whole budget would never fit, so InitialStructures passes over it with a warning and records it in oversized_rows.

A diverged trajectory is frozen on the step the predicate flags it, but the propagator keeps stepping and evaluating the whole batch, frozen graphs included, so a graph frozen with infinite forces presents a non-finite or runaway geometry to the model on every remaining step of the segment, which a model’s neighbor-list builder may not survive; quarantine_diverged (default False) zeroes the flagged graph’s velocities and forces and makes its positions finite on that step, and the sanitized frame is never captured.

The mixture ratio#

replay_ratio, written λ, is the fraction of every training batch drawn from the replay buffer; the rest comes from the reference dataset, reference_dataset. It is the setting to reach for first, because it decides how far the run is allowed to follow its own trajectories:

  • λ = 1 trains on generated data alone and takes no reference dataset. The student is pulled entirely toward wherever its own dynamics go, which is also the failure mode: if the trajectory drifts into configurations the teacher was never meant to describe, nothing pulls it back. Passing a reference_dataset anyway is rejected rather than ignored, because it would be checked for schema and device and then never sampled.

  • 0 < λ < 1 keeps a fixed, teacher-labeled distribution in every batch. The reference dataset is the pull: it holds the student on data whose coverage you chose, while the generated share keeps closing the gap between the training distribution and the one the student actually visits.

  • λ = 0 is offline distillation. The loop rejects it rather than running generation whose frames it would never train on. Drop on_policy and call run(loader) over the labeled store instead.

The composition is exact per batch rather than an average: with replay_ratio=0.25 and batch_size=16, every optimizer step sees twelve reference samples and four generated ones. The achievable granularity is 1 / batch_size, so the two settings only mean something together. A ratio that rounds either source down to zero samples per batch is rejected at construction, and the error names the smallest batch size that works.

The loop rebuilds the mixed loader every segment, because the batch sampler reads its child datasets’ lengths once and the buffer grows between segments. If you drive build_mixed_loader() yourself, do the same, or the newest frames are never sampled. Each rebuilt sampler seeds its generator with OnPolicyConfig.seed plus the segment index, so the reference draw is reproducible across runs without repeating within one. That setting, not the global torch seed, is the mixture’s randomness. Because the generator seed is a sum, two runs whose seed values differ by one draw the same sequence shifted by one segment. Replicates meant to be independent need seed values at least num_steps // training_steps_per_segment apart.

The reference dataset must be teacher-labeled#

A mixed batch is one collated Batch, and collation is not a merge. It keeps only the fields both sources hold and drops the rest, while a whole level that only one side holds is zero-filled for the other side’s samples. Either behavior would be silent, so both are rejected instead: the reference dataset’s schema is compared against the buffer’s, on a probe batch drawn from each side.

The schema the reference dataset has to match is the replay-frame contract: the structure, whatever propagator state travels with it, and the teacher_* labels. The contract excludes the energy, forces, and stress the propagator wrote on the live frame. The labeling hook strips those on the way into the buffer, so a stored frame never carries the student’s own prediction under a reference target’s name.

Warning

A raw DFT-labeled dataset cannot be used as the reference dataset. Its energy and forces are reference labels, not teacher labels, and it is refused rather than quietly mixed in. Running it through label_dataset() is necessary but not sufficient: labeling carries every source field over, so the labeled store holds teacher_energy and teacher_forces alongside the energy and forces it started with, and the check refuses it just the same.

The remedy is to strip the reference labels on the way in, because label_dataset writes what the dataset hands it and applies no transform of its own. A per-sample transform on the streaming Dataset is the general tool. A field it sets to None is gone from the batch load_batches re-forms, because from_data_list() takes its key list from the sample’s non-None fields.

from nvalchemi.data import AtomicData
from nvalchemi.data.datapipes import AtomicDataZarrReader, Dataset


def strip_reference_labels(
    data: AtomicData, metadata: dict
) -> tuple[AtomicData, dict]:
    """Drop the reference labels a generated frame never carries."""
    data.energy = None
    data.forces = None
    data.stress = None
    return data, metadata


unlabeled = Dataset(
    reader=AtomicDataZarrReader("dft.zarr"),
    device="cpu",
    transforms=[strip_reference_labels],
)
label_dataset(unlabeled, scorer, "reference.zarr", batch_size=64)
reference_dataset = Dataset(reader=AtomicDataZarrReader("reference.zarr"), device="cpu")

Assigning None is the deletion idiom, since AtomicData has no __delitem__. The transform has to return the (data, metadata) pair it was handed. Two things defeat it. skip_validation=True builds the fused batch straight from raw tensor dicts and never runs the per-sample pipeline, so leave it at its default here. And the transform has to strip every sample alike: a batch whose first sample kept forces while later ones dropped them fails collation on a batch-dimension mismatch.

When the labeled dataset fits in memory, an InMemoryDataset built with a batch_transforms= entry that deletes the three keys off the resident Batch does the same. That transform runs once, as the batch is materialized, so every chunk label_dataset reads is already stripped, and Batch, unlike AtomicData, supports del. Data Loading Pipeline covers both transform kinds. Choose between the two on memory: the streaming form has no ceiling. The third option is to label structures that never carried reference labels at all, which is what build_systems does in On-Policy Knowledge Distillation.

Two checks enforce all of this, at different points. At construction, the teacher_* field sets of the two sources are compared for equality: a reference dataset with no teacher labels, one narrower than the generation scorer, and one wider all fail alike. The dataset is also probed for any field a generated frame can never carry. That probe catches a store labeled over an existing DFT-labeled dataset before a single teacher pass is paid. The full field, level, and dtype comparison against real frames is left to the first segment’s mixed loader, so it surfaces only once a segment has generated and labeled. That is a whole segment of forward passes with an expensive teacher, so it is worth knowing the frame schema up front. A stored frame is whatever the initial structures carry, minus everything run-local, plus one field per teacher signal. The run-local fields are energy, forces, stress, the neighbor tensors, and the dynamics bookkeeping, which is whatever bookkeeping_keys() reports across the propagator’s stage tree (status and system_id among them). Structures built from plain AtomicData with energy and forces zero-filled therefore store positions, atomic_numbers, atomic_masses, atom_categories, and velocities, plus cell and pbc for a periodic system.

To read it off the run rather than off this list, take one throwaway segment with no reference dataset and compare:

probe = DistillationStrategy(
    models={"student": student, "teacher": teacher},
    optimizer_configs=optimizer_configs,
    loss_fn=loss_fn,
    num_steps=1,
    on_policy=OnPolicyConfig(
        dynamics=dynamics,
        teacher_scorer=scorer,
        initial_structures=InitialStructures(initial_dataset),
        replay_ratio=1.0,
        training_steps_per_segment=1,
        batch_size=1,
        generation_steps=1,
        label_frequency=1,
    ),
)
probe.run()
print(sorted(probe.replay_buffer.schema))

The names come back as level.field, the form a mismatch is reported in. Strip the level off each name to compare against the reference dataset’s bare field_names, or compare against the same level.field names read off a probe batch drawn from the dataset.

Mixed batches require matching dtypes across sources. Batch collation casts the trailing slice to the lead slice’s dtype, and chunk order is not fixed, so mixing float64 reference data with float32 generated frames would change precision nondeterministically. The loader therefore rejects dtype mismatches outright. One scorer alone does not give the two sides the same dtypes. A store hands every floating field back at the dtype of the dataset’s positions, which is float32 for essentially every dataset, while a generated frame keeps whatever dtype the generation scorer emitted. Build that scorer with an explicit dtype matching what the store returns, and label the reference dataset with the same scorer. The example needs no cast only because its teacher already computes at float32.

Supervising one batch against teacher labels and reference labels at once needs masked loss composition, which is not supported yet. An on-policy run is therefore supervised by the teacher throughout. Annealing between teacher and reference targets with a LossWeightSchedule is an offline technique.

Both mixture sources are collated before the strategy moves the batch, so the reference dataset’s device pins the replay buffer to it. Leaving replay_device unset stages generated frames there. Naming a different device is rejected at construction, rather than discovered mid-run as a cross-device collation failure. The device that counts is the one the dataset actually emits on, read through the core dataset_device(), with same_device() deciding whether an index-less cuda and an indexed one point at the same place (both are documented in Data module (AtomicData, Batch, readers/writers)). To move the mixture, therefore, open the reference dataset on the device the run trains on. replay_device matters only in a run with no reference dataset, where the frames stay in host memory unless it names another device.

Scaling the segment loop out#

The runbook — the DDPHook, the torchrun lines, and the multi-node rendezvous — is in Scaling out: multi-GPU and multi-node. This section covers what the segment loop asks of the settings once the world is larger than one rank.

The initial structures are dealt out strided, by shard(): rank r takes every world_size-th structure from offset r. The shards are therefore disjoint, cover the dataset, and differ by at most one structure. Those rows are all that the rank may propagate, and they are public as structure_shard. The position counts rows of that shard rather than rows of the dataset, so the initial batch and every later backfill draw from the rank’s own shard alone. system_id is not a row of that shard. The ids number the trajectories a rank has started, so under recycle they keep climbing while the position wraps, and each rank hands them out from its own base. The deal itself is the core distributed_shard(), which a source of your own can call for the same disjoint, strided share.

Two consequences follow from the deal. First, a dataset holding fewer structures than there are ranks is refused; the verdict is agreed across ranks through all_reduce_flags() before the first gradient collective, and the error names the ranks whose shard seeded nothing. Second, a dataset that does not divide evenly draws a warning rather than a refusal. Every rank draws the same number of replay samples per batch from a buffer holding only its own trajectories, and the gradients are averaged rank by rank, so a frame generated on a shorter shard reaches the optimizer with more weight. Size the dataset as a whole multiple of the world size. Striding by index balances sample count rather than atom count, so sort the dataset by atom count before sharding to balance the computational load across ranks.

The world divides the generation work rather than multiplying it. A segment’s aggregate frame count, and the teacher bill paying for it, is what the single-process run produced, with each rank contributing a 1/world_size share. generation_steps, label_frequency, and replay_capacity are all per rank. At a fixed replay_capacity, each rank’s buffer therefore spans world_size times as many segments before FIFO eviction reaches back, and every mixed batch grows staler as the world grows. Raise generation_steps or the structure count alongside the world, or lower replay_capacity by the world size. Do not do both, or the buffer spans only 1/world_size of the history the single-process run had.

Sharding separates the structures, but it does not separate the randomness on its own. The loop owns two seeded streams: the mixture sampler’s OnPolicyConfig.seed, and every integer random_seed in the propagator’s stage tree, such as random_seed. Both are moved onto a per-rank stride of the seed space, OnPolicySettings.rank_seed_stride (default 1_000_003): the propagator side goes through seed_offset() with rank * rank_seed_stride, undone with the negated offset when the run exits. Keep the stride above the run’s step count, since both streams add a step counter to their base seed. seed_offset returns the stages it could not move, such as one exposing a torch.Generator and no integer seed, and those stay on the shared stream. Every rank names them in a warning before the first segment, and you have to give them a rank-distinct seed yourself. That matters most when the initial structures are replicas of one geometry, which is how a run asks for one trajectory per rank. Sharding separates nothing there, and an unmoved stage makes every rank generate identical frames, billed once per copy.

A multi-rank launch requires a synchronized student. With a DDPHook registered, the check reads wrapped_keys and passes when "student" is among the models the hook wrapped; without one, something has to own models["student"] after SETUP in the way unwrap_model() reads it, as a gradient-synchronizing wrapper of your own does. A launch that leaves the bare student registered is refused, since each rank would otherwise train a private student and only rank zero’s would be checkpointed. A wrapper working in place, such as FSDP2’s fully_shard or hook-based gradient synchronization, leaves the registered object as it was, so the student reads as unwrapped. OnPolicySettings.require_wrapped_student=False waives the refusal with a one-time warning, and keeping the ranks’ students in step is then up to you.

Where the reference dataset sits decides where every rank collates, because the replay buffer follows it: keep it in host memory or let it emit lazily, and never stage it eagerly on one GPU before the hook has pinned the rank. Scaling out: multi-GPU and multi-node covers the placements, including the SETUP hook that moves a host-memory dataset onto the rank’s device, and the index-less replay_device that resolve_device() resolves to the device this rank has made current.

A multi-rank restart needs no device bookkeeping: restore_checkpoint() loads onto the live devices, and run() re-homes the optimizer state after the hook has pinned the rank. The propagator state is not checkpointed, so a restart reseeds every rank’s trajectories from that rank’s shard and refills the replay buffer from scratch; budget the first segments after a restart as cold (see The segment is the restart granularity under Operational notes).

Non-conservative teachers#

Some teachers predict forces from a dedicated head rather than as the negative gradient of their energy. Such a force field is non-conservative: it does not integrate to a potential energy surface, and its curl need not vanish. That is a fine trade for a labeling model and a bad one for a model driving long MD, so distilling a direct-force teacher into a student whose forces are an energy gradient is a core use case here rather than a workaround.

Nothing in the distillation path gates on conservativeness — not the strategy, not the scorer, not the losses. There is no flag to set. The scorer detaches every signal it returns, so how the teacher produced a force never reaches the student’s autograd graph; a teacher force is a number in a batch field, exactly like a DFT force from a dataset.

A conservative student cannot represent the non-conservative part at all. Its force field is, by construction, minus the gradient of a scalar, and gradient fields are curl-free. Minimizing a force-matching objective therefore drives the student toward the curl-free field closest to the teacher’s, in the least-squares sense the loss defines. The non-conservative component is projected out rather than badly fitted. That is usually what you want, since it is the component that would have shown up as energy drift in the student’s own dynamics. The force-matching loss is therefore bounded below by the teacher’s non-conservative residual on the training distribution. A nonzero plateau in force error reflects this irreducible residual rather than poor optimization.

Two practical consequences follow. Do not expect force-matching error against a non-conservative teacher to go to zero: the floor is the size of the projected component, and non_conservative_residual() measures it. And keep a total-energy term in the objective. A force-matching term only ever sees the gradient of the student’s energy, so forces alone fix the student’s energy only up to an additive constant.

Evaluating the student#

Acceptance is a handful of measurements and one verdict formed from them. A student is accepted when it clears every acceptance bar it is given; an acceptance bar is a limit on one number from the student’s measurements, such as a maximum force MAE. Import the measurements from the evaluation subpackage rather than from the distillation namespace: an acceptance run pulls in the dynamics engine and the reporting stack that training itself does not need.

evaluate_accuracy() scores a held-out set against the dataset’s own labels or against the teacher’s, whether the teacher’s labels are on disk or scored on the fly. The quantities it compares are an open table. Each built-in quantity is an AccuracyQuantitySpec in BUILTIN_ACCURACY_QUANTITIES, which names the prediction and reference keys, the teacher signal that labels it, and the supervised loss that scores it. evaluate_accuracy(quantities=[...]) takes those names, or a spec of your own for a head the table lacks, as long as one quantity in the set carries a supervised loss. label_dtype= pins the dtype that scored-on-the-fly labels are cast to. A student behind a data-parallel wrapper is read through unwrap_model() when the gradient mode is decided.

Accuracy alone is not what a small student fails at, so stability is measured on a trajectory the student drives itself. Register a StabilityMonitor on the propagator, and read monitor.metrics() once the run is over. metrics() is a method, not an attribute, and it needs two samples at two different steps. The monitor has three settings. divergence is the predicate that decides which frames count as diverged (the loop’s nonfinite_divergence by default), and the step it first fired on is reported as first_divergence_step, which the opt-in no_divergence bar fails a student on. aggregate chooses whether the reported drift is the worst graph in the batch ("max", matching the energy-drift monitor hook) or the mean over graphs ("mean"). stop_on_composition_change, on by default, stops the measurement with a warning when the batch composition changes.

extensivity_error() checks that energy still scales with replicated cells, built with the core make_supercell(). Pass extensive_keys and intensive_keys through extensivity_error for a system field of your own, since one named in neither is refused rather than guessed at, and drop_keys= for the bookkeeping a sampler-seeded run stamped. Periodicity is read from pbc before the presence of a cell. The radial-distribution pair, radial_distribution and compare_radial_distributions, compares the structure a trajectory samples against a reference trajectory’s. A student distilled from a direct-force teacher also needs non_conservative_residual(), which bounds how well any conservative student can fit that teacher (see Non-conservative teachers).

Those measurements go into one StudentEvaluation per candidate. State the acceptance bars as AcceptanceThresholds and hand both to build_acceptance_report(), which returns a report that renders as Rich tables, exports as a plain dictionary, and says whether the student is accepted:

from nvalchemi.training.distillation.evaluation import (
    AcceptanceThresholds,
    StudentEvaluation,
    build_acceptance_report,
    evaluate_accuracy,
    measured_bars,
)

evaluation = StudentEvaluation(
    name="small",
    accuracy=evaluate_accuracy(student, holdout, targets="teacher", scorer=teacher),
    stability=monitor.metrics(),
    weights="raw",
)
thresholds = AcceptanceThresholds(
    max_forces_mae=0.05, max_energy_drift_per_atom_per_ns=0.005
)
print(sorted(measured_bars("accuracy", "stability", accuracy_quantities=("energy", "forces"))))
report = build_acceptance_report([evaluation], thresholds)
print(report.accepted)

A bar with no measurement behind it fails the student rather than being skipped, so state only the bars the measurements in hand can decide. Ask measured_bars() which those are, instead of restating the mapping in your own script. It takes the measurement families that were filled, plus the quantities an accuracy pass actually compared. The from-scratch bar, max_from_scratch_ratio, needs a baseline_accuracy scored on the same holdout by an equal-size student trained from scratch. The ratio is taken on energy_per_atom_mae, forces_mae, and stress_mae, whichever both carry, and the worst one is kept.

The bars form an open table rather than a closed list. Each is an AcceptanceBar that names the threshold it is set under, the measurement families it reads, the check, the accuracy quantities that decide it, and the direction it passes in. The shipped bars are DEFAULT_BARS, and BAR_FAMILIES is derived from them. A bar of your own is a limit on a number you file under StudentEvaluation.extra. Put it in a table handed to build_acceptance_report(..., bars=) and measured_bars(..., bars=), and set its limit under its name in AcceptanceThresholds.extra. Every measurement is a Pydantic model sharing MeasurementRecord, so to_dict and from_dict carry those open slots along with the built-in fields. AccuracyMetrics.errors holds the quantities the built-in fields do not name.

weights is not a measurement. It records which of the student’s two weight sets the numbers came from, "ema" or "raw". Nothing downstream can infer it, so set it here; two exports of the same student then say which artifact each one gated on. None records nothing, which is not the same as "raw". evaluate_accuracy scores exactly the object it is handed, with no EMA swap in either direction. A student trained under an EMAHook therefore has to be handed over as strategy.inference_model["student"] and recorded as "ema"; passing strategy.models["student"] gates on weights that will not ship. The inference_model slot survives no checkpoint. A reloaded strategy holds nothing to score there until its run() dispatches SETUP, which republishes the averaged copy from the re-attached hook; before that, read the average off the hook itself with get_averaged_model().

Read StabilityMetrics.energy_fluctuation_per_atom and max_energy_excursion_per_atom beside a drift number, rather than reading the drift alone. A drift rate is the slope of a least-squares line. The fluctuation is the RMS residual about that fit, so a drift no larger than the fluctuation is a line drawn through an oscillation rather than a trend. The excursion says how far the series went in the meantime. Neither is a bar: the stability family gates on max_energy_drift_per_atom_per_ns, max_energy_drift_per_atom_per_step, max_momentum_drift, and the no_divergence flag alone.

The radial-distribution comparison is continuous in the positions. Pairs are deposited cloud-in-cell into bins, from a neighbor list built one bin past r_max. A rigid translation of a crystal therefore scores a Jensen-Shannon divergence at round-off, where a hard-edged histogram reports a few times 1e-2. That is what makes the metric usable on relaxed and crystalline frames. r_max may exceed half the shortest cell vector.

Custom components#

Six extension points of the distillation loop are protocols rather than base classes. A component of your own is any object satisfying the protocol; the Distillation API reference documents the distillation ones in full, and API Reference the core ones.

  • TeacherScorer — signals and label(batch) returning {teacher_field: (detached tensor, level)}; declare label_fields so the fields you write are known before the first batch. A teacher output the built-in table lacks rarely needs a scorer of its own: a TeacherSignal passed to the in-process scorer covers it. A scorer is a live object that no spec carries, so a rebuilt run is handed its OnPolicyConfig again.

  • initial_structures — any object satisfying the core StructureSource protocol (see API Reference), importable from the distillation package as InitialStructuresSource. A source driving a relaxation lifecycle stamps status zeros and system_ids on the batch it hands over, as the core OrderedStructureSampler does.

  • The fits= policy of a draw you drive yourself — any object satisfying the core FitPolicy protocol; WithinBudget is the stock one. Both are re-exported from the distillation package.

  • AdmissionPolicy — a callable over a batch of captured frames returning one boolean per graph; frames it refuses never enter the replay buffer.

  • EvictionPolicy — select(buffer, incoming, capacity) returning the indices into the resident batch (oldest first, the admitted frames last) to drop, at least as many as the buffer is over capacity by; FIFO is the reference.

  • capture_sink — any DataSink. The loop sizes it to (generation_steps + 1) frames per trajectory, growing it through resize(capacity) when the sink also satisfies the core ResizableSink protocol (see API Reference); a smaller sink that does not is refused. GPUBuffer is the in-tree device-resident one.

Minimal implementations of three of these protocols follow, wired into one loop with a device-resident capture sink:

import torch

from nvalchemi.dynamics.integrators.nvt_langevin import NVTLangevin
from nvalchemi.dynamics.sinks import GPUBuffer
from nvalchemi.training.distillation import OnPolicyConfig, TeacherScorer


class TabulatedScorer:
    signals = frozenset({"energy"})
    label_fields = ("teacher_energy",)

    def label(self, batch):
        return {"teacher_energy": (torch.zeros(batch.num_graphs, 1), "system")}


def finite_labels(frames):
    return torch.isfinite(frames.teacher_energy.view(-1))


class DropNewest:
    def select(self, buffer, incoming, capacity):
        return torch.arange(capacity, buffer.num_graphs, device=buffer.device)


assert isinstance(TabulatedScorer(), TeacherScorer)
config = OnPolicyConfig(
    dynamics=NVTLangevin(student, dt=0.5, temperature=300.0),
    teacher_scorer=TabulatedScorer(),
    initial_structures=dataset,
    capture_sink=GPUBuffer(capacity=4096, max_atoms=64, max_edges=0, device="cuda"),
    replay_admission=finite_labels,
    replay_eviction=DropNewest(),
    replay_ratio=1.0,
    training_steps_per_segment=32,
)

Operational notes#

Validation data is labeled on the fly too. On-the-fly labeling runs on BEFORE_FORWARD, a stage that both the training loop and the validation loop dispatch on the device-placed batch. Unlabeled validation data therefore needs no preparation, and a caller-supplied training_fn is covered too. Pointing validation_config at a store written by label_dataset still avoids the teacher pass entirely. Wrap that store in a DataLoader, or any iterable of Batch, before handing it to ValidationConfig; a bare Dataset iterates (AtomicData, metadata) pairs rather than batches. every_n_epochs=1 validates at every segment boundary, and every_n_steps fires inside segments.

Composed weights are ratios, not coefficients. ComposedLossFunction renormalizes weights by default, so the three-term objective above runs at 1/2.2, 1/2.2, and 0.2/2.2. normalize_weights=False keeps them literal, which also stops a weight schedule on one term from rescaling the others (see Losses).

Label dtype follows the student, floored at single precision, unless label_dtype says otherwise. By default, teacher labels are cast to the dtype of the student’s first floating-point parameter, so a float64 teacher feeds a float32 student without a dtype error at the loss. The cast never goes below float32. A store hands every floating field back at the dtype of the dataset’s positions, and a narrower label would disagree with what label_dataset persisted. A bfloat16 or float16 student therefore needs dtype_policy="prediction_to_target" on its loss terms. A float64 student training from a store needs "target_to_prediction", which widens the float32 labels a store returns. The cast is resolved at construction, so a student whose dtype changes afterwards also needs a dtype_policy, or an explicit DistillationStrategy(label_dtype=torch.float64), which names the dtype outright. A non-floating label_dtype is refused.

Checkpoints serialize every model, teacher included. A periodic CheckpointHook write therefore costs the frozen teacher’s weights as well as the student’s, so size the interval for a large teacher.

The segment loop is runtime-only. to_spec_dict() carries the declarative settings and omits on_policy and reference_dataset with a warning, because they hold a live propagator, scorer, and datasets that no spec can describe. A strategy rebuilt from such a spec runs offline until they are supplied again: from_spec_dict, from_checkpoint_dict, and load_checkpoint all take on_policy= and reference_dataset= as keyword arguments, and the live objects passed there are the ones the rebuilt run uses.

The segment is the restart granularity. A checkpoint carries the counters, the weights, the optimizer state, and the checkpointable hook state, but not the propagator state. A resumed run therefore reseeds its trajectory from the front of its shard and refills the replay buffer from scratch, and a segment that a checkpoint interrupted part-way is counted as finished on the way in. Its AFTER_EPOCH hooks never fire, the batches it had left are not replayed, and the run opens a fresh segment, which begins by generating. The buffer does outlive a run within one process. A second run() on one strategy appends to the frames the first run filled, but it reseeds its own trajectory, because installing the rank shard rewinds the position.

Resuming an on-policy run has two routes. The first is load_checkpoint(), which rebuilds the strategy from the spec the checkpoint carries. Pass models= so the propagator is rebound to the very student the optimizer updates, and on_policy= and reference_dataset=, which the spec cannot carry. Without on_policy the strategy comes back offline-shaped, and its run() then rejects the None dataloader. The second route is to rebuild the strategy with the same propagator, scorer, reference dataset, and hooks, then restore the counters, weights, optimizer state, and checkpointable hook state into it in place with restore_checkpoint(). Unlike load_checkpoint, it takes no hooks override, because it restores into the hooks the rebuilt strategy already holds:

from nvalchemi.training.hooks import CheckpointHook

strategy = DistillationStrategy(
    models={"student": student, "teacher": teacher},
    optimizer_configs=optimizer_configs,
    loss_fn=loss_fn,
    num_steps=20,
    on_policy=on_policy,
    reference_dataset=reference_dataset,
    hooks=[CheckpointHook("runs/on_policy/checkpoints", step_interval=3)],
)
strategy.restore_checkpoint("runs/on_policy/checkpoints")
strategy.run()

Attaching on_policy to an already-loaded strategy is not a substitute. Assignment is not validated, so the propagator would keep a student the optimizer never updates, and the run would silently stop being on-policy. num_steps is an absolute target rather than a budget for the resumed leg, so a run that already reached it resumes to nothing until the target is raised.

Note

Reserved and runtime-only settings. OnPolicySettings.replay_eviction admits one spelling, "fifo"; a custom EvictionPolicy instance rides on OnPolicyConfig alone, as replay_admission, divergence, capture_sink, and convergence_hook do, and the detached OnPolicyConfig.settings copy records it as "fifo" with a warning. weight_sync_frequency must be 1: the propagator and the trainer share one module object, so an eager run is never out of sync, and the setting becomes meaningful only once the propagator holds a compiled or remote copy of the student.

API reference#

See Distillation API for the reference documentation of DistillationStrategy, InProcessTeacherScorer, label_dataset(), OnPolicyConfig, OnPolicySettings, nonfinite_divergence() (an alias of the core nonfinite_graph_mask()), InitialStructures, TeacherLabelHook, ReplayBuffer, AdmissionPolicy, EvictionPolicy, FIFO, AtomicEnergyMatchingLoss, EmbeddingMatchingLoss, HessianMatchingLoss, and BoltzmannMatchingLoss. It also covers the core OrderedStructureSampler and StructureSource, and the core FitPolicy, WithinBudget, and ResizableSink that the distillation package re-exports.