Distillation API#
Teacher scoring, offline dataset labeling, the offline distillation strategy and loss terms, and the on-policy generation components for knowledge-distillation workflows.
See also
User guide: Distilling a Teacher Into a Student
Training strategy API: Training strategy API
Fine-tuning API: Fine-tuning API
Loss API: Losses — Training Terms
Scoring#
A scorer turns a Batch into named teacher signals,
each a TeacherSignal mapping one
teacher output to a batch field, a level, and a canonical shape. The built-in
ones — energy, forces, stress, atomic_energies,
embeddings, and hessian — are requested by name; any other teacher
output is requested as a spec of its own, passed beside the built-in names:
from nvalchemi.training.distillation import InProcessTeacherScorer, TeacherSignal
charges = TeacherSignal("charges", "charges", "teacher_charges", "node")
scorer = InProcessTeacherScorer(teacher, ["energy", "forces", charges])
scorer.label_fields # ('teacher_charges', 'teacher_energy', 'teacher_forces')
The spec reads the teacher’s charges output into the node-level
teacher_charges field; a normalize callable reshapes a raw output whose
layout differs from the field’s, and the scorer refuses the spec at
construction when the teacher does not declare that output.
InProcessTeacherScorer evaluates a
teacher loaded in the current process and leaves the scored batch exactly as it
found it, including neighbor tensors. Label precision is the scorer’s decision.
Nothing that calls a scorer opens an autocast region of its own, so a scorer
runs inside whatever autocast region is open at the call site. The in-process
scorer’s autocast setting picks the mode. The default False disables
autocast for the scoring pass, so a mixed-precision region around the call
never reaches the teacher. None leaves the caller’s region in force. A
floating-point dtype enables autocast at that dtype, whether or not a region is
open. True enables it at the autocast dtype in force for the device: the
device default when no region is open, the caller’s region’s dtype when one is.
Structural interface for objects that produce teacher signals for a batch. |
|
Score a batch with a teacher model loaded in the current process. |
|
Model output, batch field, level, and shape rule behind one teacher signal. |
|
Return every batch field the given signals populate, sorted. |
|
Return the batch fields scorer populates, or |
|
Return the signal that populates field, or |
|
- nvalchemi.training.distillation.BUILTIN_SIGNALS: Mapping[str, TeacherSignal]#
The built-in
TeacherSignalspecs, keyed by signal name.
- nvalchemi.training.distillation.SUPPORTED_SIGNALS: frozenset[str]#
The names of the built-in signals: the keys of
BUILTIN_SIGNALS.
- nvalchemi.training.distillation.TeacherLabels: TypeAlias#
Teacher signals for one batch, keyed by the batch field they populate:
dict[str, tuple[torch.Tensor, SignalLevel]].
Scorers use three public type aliases. SignalLevel is the "node" or
"system" level a signal is attached at. TeacherLabels is the
{batch field: (detached tensor, level)} mapping that
label() returns.
SignalNormalizer is the type of the callable a spec’s normalize slot
takes. The built-in specs are published as
BUILTIN_SIGNALS, keyed by name, and
their names as SUPPORTED_SIGNALS. A
TeacherSignal names the teacher
output it reads, the teacher_* field it writes, the level, and an optional
normalize callable shaping the raw output. When a spec declares companion
extra_fields, its normalize returns a mapping over all of its
fields instead of a single tensor, and the spec is refused without a
normalize. The namespace and level rules are enforced when the spec is
built, and the in-process scorer refuses a spec naming an output the teacher
does not declare. The scorer publishes its
resolved specs as signal_specs and the fields they write as
label_fields. A custom scorer may publish label_fields, the batch
fields its label() populates, which consumers resolve through
scorer_fields() rather than reading the
attribute.
Where the teacher’s neighbor list comes from is an explicit setting,
neighbor_list. The default "rebuild" builds the teacher’s own list for
every call and rolls it back afterwards, whatever list the batch carries; a
composed pipeline keeps its default source’s list as an instance attribute and
captures its whole per-source table alongside it, and both are hidden from the
teacher for the duration of scoring, so a teacher scoring a live student batch
never reads the student’s neighborhoods. "reuse" is for the case where the
student has already built the list the teacher needs, in the teacher’s format
and at its cutoff: the scorer consumes the batch’s list and builds nothing,
checking only what it cannot infer — that the keys the teacher’s format reads
are present and that a cutoff stamp, if the batch carries one, equals the
teacher’s — and raising a ValueError naming the missing key or the
mismatched cutoff otherwise, never falling back to a rebuild. Whether a list
holds each pair once or twice is recorded nowhere on the batch, so a reused
list must match the teacher’s half_list by construction. A teacher
composition that plans more than one neighbor-list source is refused at
construction, because the scorer builds exactly one list per batch; compose it
with neighbor_adaptation="always" or a max_cutoff_ratio of at least its
largest-to-smallest cutoff ratio so it adapts that one list per step.
A composed teacher also wires one stage into the next through the batch: an
intermediate such as charges is written straight onto it, and an autograd
group swaps each of its gradient inputs for a fresh leaf. The scorer records
the batch’s fields before the forward pass and afterwards drops the ones that
appeared and puts back the ones that were replaced, so a teacher never leaves
its charges, or a positions tensor cut loose from the student’s graph, behind
for a later student forward to read.
Forward-pass signals share one teacher pass, and embeddings adds a second.
hessian labels a Hessian-vector product: the product of the teacher’s
energy Hessian with a random probe direction, computed without forming the
Hessian. It adds an energy-only pass plus the backward passes that
HessianOperator takes through it: the first
position derivative, a connectivity probe of that derivative, and the
product itself. hessian is the only built-in signal that writes two
fields: the product
in teacher_hvp and the probe direction in teacher_hvp_probe. The
student is later differentiated along that same probe.
label_hvp()
computes one product for a probe the caller chooses. probe_seed pins the
probe direction. Left unset, every labeling draws a fresh direction, which is
what covers the Hessian over a run.
DistillationStrategy sets it per
validation batch, so that the validation metric is comparable across passes.
Labeling#
Offline labeling walks a dataset once, scores it, and writes the source fields
plus the teacher fields to a Zarr store that the ordinary reader and dataset
path consume. Runs are resumable: the first len(store) samples are skipped,
a store that already covers the dataset is a no-op, and a store holding more
samples than the dataset — one written from a different dataset — is refused.
Every chunk must write the fields, levels, dtypes, and row shapes the store
holds, since the writer would otherwise misalign, cast, or truncate labels
without an error, and a store whose arrays disagree about how many samples it
contains — what an interrupted run leaves behind — is reported rather than
resumed from a misaligned offset. Both checks read the store through the
reader’s own description of it:
check_integrity() refuses the torn
store, and schema() supplies the
per-field FieldSchema each chunk is compared to. Each
label is held to the chunk’s atom or graph count before it is attached, because
the split into per-graph rows would otherwise drop whatever a scorer returned
beyond it.
The neighbor tensors are dropped by default. The dense ones cannot append into
a fixed-width store array, and a sparse list is dropped because the cutoff it
was built at is a batch attribute the store does not hold, so a reloaded list
is one nothing downstream can check; keep_neighbors=True stores the sparse
list anyway. Build the student’s list from the stored positions with a
NeighborListHook at BEFORE_FORWARD. Labels may
be stored in any dtype an ALCHEMI store holds (dtype on the scorer picks
it), but they read back at the reading dataset’s positions dtype, because a
dataset coerces every floating-point field it loads; the stored dtype governs
the store’s size, not what training sees.
Labels are written with overwrite=True, so a scorer that reached outside the
teacher_* namespace would replace the reference field of that name and
persist the replacement. A scorer’s declared label_fields is refused before
the first chunk is written, and the fields each chunk actually returns are
refused again per chunk, which is what polices a scorer that declares nothing.
The chunk loop can read one chunk ahead of the scoring and writing of the
previous one, through the dataset’s fused-prefetch surface
(prefetch_fused_batches / get_fused_batches). Reading ahead saves up
to one load per chunk when the store is slow to read, such as a network or
object store, or when per-sample validation dominates the load; on a fast
local store the prefetch thread competes with the main thread while the
teacher’s kernels are launched, and labeling can run a little slower than the
sequential loop. prefetch="auto" (the default) therefore measures rather
than assumes: it reads the first two chunks sequentially, reads ahead only
when the second chunk’s load took at least half of its scoring and writing,
and falls back to sequential reads if the first chunk read entirely ahead was
not faster per atom than the sequential one. True always reads ahead (a
dataset without the surface falls back with a warning) and False keeps
the sequential loop. The per-chunk writes, the resume bookkeeping, and the
store’s contents are the same in every mode. A dataset that emits
host-resident chunks, with device passed to label_dataset for the
move, keeps the device transfer on the main thread and reads ahead faster
than one that transfers from the prefetch thread.
Label dataset with teacher signals and persist the result to store. |
Strategy#
DistillationStrategy is a
TrainingStrategy over the named models
"student" and "teacher". The teacher is frozen by omission from
optimizer_configs, the teacher signals are derived from the teacher_*
targets the loss reads, and batches that arrive unlabeled are labeled on the fly
unless label_missing=False skips the teacher and lets the missing target
surface from the loss. training_fn stays a plain student forward, defaulting
to default_distillation_fn(), whose
predicted_* keys are checked at construction against the outputs the student
actually computes — its active_outputs intersected with its declared
outputs — so a student whose active set is narrowed is caught before the run
rather than on its first batch. A teacher_* target that no built-in signal
populates — a field a custom scorer wrote through label_dataset — is read
from the batch as it arrives: it is neither derived into a signal nor attached
on the fly, so a batch lacking it surfaces as a missing loss target.
A validation_config carrying its own loss_fn takes part in both checks:
its teacher_* targets widen the derived signal set, and its prediction keys
are checked the same way whenever the effective validation function
(validation_fn falling back to training_fn) is the stock one. Neither
re-runs on assignment, so pass validation_config to the constructor — or,
when rebuilding from a spec, to from_spec_dict, which takes it as a runtime
override because specs exclude it — or name the wider set in
teacher_signals. Every resolved signal — derived or
explicit — is a request for its fields on every batch: a batch counts as
labeled only when it carries every resolved field, so adding a validation loss
with a new teacher_* target puts a training store written before it back on
the teacher, batch after batch, at identical values.
Training and validation batches go through one labeling seam: an internal hook
on BEFORE_FORWARD, a stage both loops dispatch on the device-placed batch.
The strategy’s own scorer disables autocast, so mixed-precision training does
not change the targets, and an on-the-fly label matches the offline one exactly
wherever the store returns the label dtype (see Labeling above): over the usual
float32 dataset every student but a float64 one agrees on both paths, while a
float64 student reads float32 back and needs a dtype_policy. Labels are
never cast below single precision, so a bfloat16 or float16 student gets
float32 labels and needs dtype_policy="prediction_to_target" on its loss
terms; label_dtype overrides that inference with an explicit floating-point
dtype. The first batch the seam labels raises one UserWarning naming the
missing fields, since every later batch without them costs a teacher pass too.
Pointing validation_config at a store written by
label_dataset() still avoids the teacher
pass entirely, and validating an EMA-averaged student against the live teacher
is ValidationConfig(use_ema="auto"), reported as model_source="mixed";
use_ema="always" currently also demands an inference-slot entry for the
frozen teacher and fails at the first validation pass without one.
The seam’s work is callable directly:
attach_teacher_labels()
attaches the teacher_* fields a device-placed batch is missing and reports
whether the teacher ran. It is idempotent, so pre-labeling a batch that later
reaches run() costs one teacher pass rather than two; a batch carrying only
some of the required fields is re-scored in full, since a partial set was
written for a different signal set than the objective reads.
Checkpoints serialize every entry of models, so each write duplicates the
frozen teacher’s weights; size the checkpoint interval accordingly with a large
teacher.
Train a student against a frozen teacher's signals. |
|
Run the student forward pass and prefix output keys with |
Two objectives need a prediction that the student’s forward pass does not
return, and each ships the training function that produces it. Both functions
are module-level, so a recipe that uses one still survives
to_spec_dict().
Both are additive: they return the stock predicted_* outputs plus one key.
embedding_distillation_fn() runs the
student’s compute_embeddings and routes the result through the
"projector" model when one is registered.
hessian_distillation_fn() differentiates
the student’s energy twice along the labeled probe. A recipe that needs both
predictions writes one module-level function of its own. Calling both stock
functions runs the student forward pass twice; building the union from
HessianOperator.from_energy
on the student’s energy and the student’s compute_embeddings avoids that.
Run the student forward pass and add its node embeddings as a prediction. |
|
Run the student forward pass and add its Hessian-vector product. |
On-policy generation#
On-policy distillation trains the student on frames it generated itself. The
run is a loop of segments. Each segment has a generation phase and a training
phase. In the generation phase, the student’s own propagator advances a batch
of structures, and the teacher labels selected frames. The labeled frames
accumulate in a replay buffer. In the training phase, each training batch is
a mixture of samples from the replay buffer and from a reference dataset, a
teacher-labeled store, split at a fixed ratio, replay_ratio.
OnPolicyConfig describes one segment
loop. It sets which propagator generates, how many steps a segment runs, how
often the teacher labels, and how much of each training batch is replayed. The
propagator is any BaseDynamics, so a
relaxation optimizer generates paths exactly as an integrator generates
trajectories. The scalar settings live in
OnPolicySettings. That class validates
on its own, so a recipe’s settings can be checked before a teacher is built.
The initial structures are the structures the generated trajectories start
from. They come from any StructureSource, the
protocol that lists the members the loop reads. The protocol is also importable
here under its historical name, InitialStructuresSource.
InitialStructures is the reference
implementation. The sampler behind it lives in the core package:
OrderedStructureSampler serves a dataset’s rows in
order, over the rows one rank owns, from a single position, next_row. The
initial batch and a restart share that position. InitialStructures adds
only the recipe round-trip. A bare dataset is wrapped in an
InitialStructures, and an object that is neither a source nor a dataset is
refused with an error that names the protocol.
draw() serves the structures.
It admits each candidate through one FitPolicy, a
predicate over the running atom and edge totals;
WithinBudget bounds those totals. At the first
candidate that does not fit, draw either stops or skips it. Stopping is how
an initial batch is packed. Skipping lets a backfill fill the room a graduated
structure freed.
Construction checks one row of the initial structures in two ways. First,
check_initial_batch() checks the row for
every field the propagator updates in place from its first step. Second, the
propagator’s compute() runs once on the row, with the same isolation the
scorer uses: evaluation mode and requires_grad flags are restored
afterwards, and the propagator’s last outputs are put back. This second check
catches declarations that no longer match the implementation, such as a
__needs_keys__ output the student never produces or a field compute()
reads that nothing declared. The error names the offending key, and it is
raised where the config is built rather than on the first step of a long run.
The check costs one student forward pass, which front-loads the kernel and CUDA
initialization the first step pays anyway.
A graph model is probed with the neighbor list its neighbor_config
declares. The list is built on the row and rolled back afterwards, so the probe
needs no hook. A model that plans more than one neighbor-list source is not
probed, and a warning says so: the probe builds exactly one list, and the check
must not refuse a propagator the loop can run. probe=False skips the
forward pass entirely. Use it for a propagator whose compute() must not run
outside the loop, or for a recipe check that should not pay for a forward pass.
Settings and live objects of one on-policy distillation segment loop. |
|
Declarative settings of one on-policy distillation segment loop. |
|
An |
The ResizableSink protocol that a capture sink
may satisfy is re-exported from this package and documented with the dynamics
sinks.
Three settings need care when sizing a run.
label_frequency is the throughput setting, because the teacher is the
expensive model. It counts against the propagator’s cumulative step_count,
so the labeling cadence does not restart at a segment boundary. Each segment
also labels the frame it ends on, which is the most on-policy frame it
produced. Both labels are keyed on the pre-increment step count: the cadence
fires at AFTER_STEP before the counter advances, and the forced label of
the last frame is keyed at step_count - 1 once the segment has run. The
next segment’s first cadence dispatch lands one step after that forced label
and is therefore skipped rather than paid for twice. With generation_steps
a multiple of label_frequency, each trajectory is labeled
generation_steps // label_frequency times per segment, which is once per
segment only when the two are equal. The first segment pays one more, for the
cadence dispatch at step 0.
replay_capacity is enforced by FIFO eviction, which drops whole frames in
arrival order. A segment contributes one frame per trajectory per labeled step.
A capacity that is not a multiple of the trajectory count therefore cuts a
segment’s frames partway through a labeled step. The structures at the back of
the batch are then over-represented in every mixture drawn afterwards. Size the
capacity as a multiple of the trajectory count.
seed seeds every segment’s mixture sampler, which adds the segment index to
it. Runs with consecutive seeds therefore draw the same mixtures, shifted by
one segment. Replicate runs draw independently only when their seeds are at
least num_steps // training_steps_per_segment apart.
weight_sync_frequency is reserved at 1. The propagator shares the
student module, so an eager run is never out of sync.
The labeling hook, TeacherLabelHook,
labels frames inline during generation. It is an AFTER_STEP dynamics hook.
It attaches teacher_* fields to the frame the propagator just resolved, each
at the level its signal declares. It can also copy a stripped version of the
frame into a DataSink. The hook never
touches the energy and forces the student wrote on the live batch,
because they drive the next step. It does strip them from the copy, together
with the neighbor tensors and the dynamics bookkeeping. A stored frame is
therefore a training sample rather than a propagator state, and it never
carries the propagated model’s own prediction under a reference target’s name.
Do not confuse the labeling hook with the strategy’s private
BEFORE_FORWARD labeling seam, which labels batches on their way into a
training step.
Labeling is idempotent per propagator step. A scorer that publishes
label_fields, or whose signal names are all built-in, is skipped when a
step it already labeled is dispatched again. A scorer that publishes neither is
skipped from its second dispatch on, once the first dispatch has revealed what
it writes. A forced label is never skipped, so the last frame of an
early-exiting segment and the final frame of the run are always labeled.
The segment loop registers the labeling hook itself. The hook stages each
segment’s frames in a capture sink, which the loop drains into the replay
buffer at the segment boundary. The capture sink is host memory by default, or
the DataSink passed as
OnPolicyConfig.capture_sink. A GPUBuffer
keeps the staged frames on the generation device and avoids a device-to-host
copy per labeled frame. The loop sizes the capture sink. A segment captures at
most one frame per trajectory per labeled step, including the forced last
frame, so the sink must hold (generation_steps + 1) frames per trajectory
in the propagated batch. A configured sink with less capacity is grown through
resize(capacity) when it satisfies
ResizableSink, and refused otherwise. A sink that
still holds frames when a segment starts is refused rather than drained as
generated data. Like dynamics and teacher_scorer, the capture sink is
runtime-only.
Label the live propagator frame with teacher signals, inline. |
|
Flag the graphs of batch holding a NaN or infinite value under any of keys. |
Generated frames land in the replay buffer, a
ReplayBuffer: an in-memory dataset
with a frozen key schema. The schema is frozen because appending a batch keeps
only the keys both sides hold, so one unlabeled frame would otherwise strip
teacher_* from every frame already stored.
build_mixed_loader() then draws each
training batch with an exact reference/replay composition, resolved to whole
samples of the batch size. Rebuild the loader after every segment, because the
batch sampler reads the child dataset lengths once, at construction.
The two sources must agree on their whole batch schema. The schemas are
compared on a probe batch drawn from each side, not on field names, because a
Zarr-backed store and an in-memory buffer report field names differently.
Collation drops a field that only one side holds and zero-fills a whole level
that only one side holds. It also casts the second part of a mixed batch to the
dtype of the first, and either source may lead a chunk. All three kinds of
difference are therefore rejected. The reference dataset must be
teacher-labeled and stored in the replay-frame shape: the structure, the
propagator state, and the teacher_* labels. A reference dataset that
carries its own reference energy or forces is rejected, because mixing
it in would silently lose or fabricate those labels. Supervising one batch from
teacher labels and reference labels at once is not supported yet; it needs
masked loss composition.
The framework owns the schema freeze and the mixture. What enters the buffer
and what leaves it are left to policy. An
AdmissionPolicy is a predicate over
the incoming frames that returns one boolean per graph. It runs before the
schema check, so the frames it rejects never enter. Examples are the
NaN-labeled frames of a diverged trajectory, or frames that fail a size or
diversity filter. An EvictionPolicy
runs after the admitted frames are appended. Given the resident batch, ordered
oldest first with the admitted frames last, and the capacity, it returns the
indices to drop. It must name at least as many frames as the buffer is over
capacity. FIFO is the reference
eviction policy. The string "fifo" builds it; that string is the only value
ReplayEviction admits, and the one a recipe stores. The segment loop passes
OnPolicyConfig.replay_admission and replay_eviction to the buffer it
owns. A policy instance on either field is runtime-only, and the declarative
settings record a custom eviction policy as "fifo" with a warning.
Hold generated frames for replay under one frozen key schema. |
|
Decide which incoming frames enter a |
|
Choose which frames leave a |
|
Eviction policy that drops the oldest frames first. |
|
Build a loader that draws a fixed reference/replay mixture in every batch. |
Setting on_policy on the strategy combines these pieces into a run.
run() then takes no
dataloader. It seeds a state batch from initial_structures and repeats
generate-label-train segments until num_steps optimizer steps are done. The
1 - replay_ratio share of every batch is drawn from reference_dataset.
A reference dataset is required unless the ratio is 1, and refused when the
ratio is 1, because it would then be checked but never sampled.
The initial batch is stamped with fresh dynamics bookkeeping on the way in.
Structures loaded from a store that an earlier relaxation graduated therefore
do not arrive frozen at exit_status. At construction, the strategy probes
the reference dataset once to check three things: the fields the labeling hook
strips, the device the dataset emits on, and the teacher fields the
propagator’s scorer declares. A mismatch in any of them is a guaranteed mixture
failure that would otherwise surface only after a whole generation segment had
been paid for.
One segment is one epoch. AFTER_EPOCH and epoch-cadence validation
therefore run at segment boundaries, and step-cadence validation runs inside
segments. The run’s closing validation is skipped when a cadence already
validated at the final step. The segment is also the restart granularity. A
run resumed from a checkpoint taken mid-segment counts that segment as finished
rather than replaying the batches it had left. The same holds for an offline
run that continues on-policy from a partial epoch. A second call to run()
on the same strategy keeps the replay buffer the first call filled and reseeds
only the trajectory. Installing the rank shard resets the sampler to the front
of its rows, so the second call generates from the same structures again, not
from whatever the first call left over.
Under a DDPHook the loop runs data-parallel. Each rank propagates its own
shard of the initial structures; see Scaling out: multi-GPU and multi-node.
Generated frames are drained to host memory and then staged on the reference
dataset’s own device, so a GPU-resident reference dataset and the replay buffer
collate on one device. replay_device overrides that choice and is checked
against the reference dataset at construction. The reference device is the one
the dataset actually emits batches on, read off a batch whenever no declaration
settles it. A MultiDataset
declares no device, and a store opened without a device declares an index-less
cuda that names whichever device is current.
The student is held in evaluation mode to generate and switched to training
mode for the training phase only. Generated frames therefore cost no
second-order graph and no moving batch-norm statistics. A propagator model that
merely composes the student is held in evaluation mode for the whole loop and
moved whole to the generation device. The training phase runs
models["student"] rather than the composition, and only the named models
move with the strategy. The propagator must hold the very module registered as
models["student"], either on its own or composed into a larger model. That
object identity is what makes each segment generate from the weights the
previous segment trained, and it is checked at construction.
Splitting a built-in propagator’s run into segments is exact. run never
resets step_count or the integrator state, and the Langevin thermostat
draws from a counter-based generator keyed on the cumulative step count. Two
segments of K steps therefore reproduce one run of 2K steps. A
dynamics hook that is sensitive to being opened and closed, such as
LoggingHook, is re-entered once per
segment. The progress of a segment that converges and
exits early is read from dynamics.step_count rather than assumed to be
generation_steps. A FusedStage pays its
priming forward pass once per segment, so prefer a bare propagator.
A custom teacher_* field that the propagator’s scorer writes is an ordinary
loss target, exactly as in offline distillation. Generation writes it on every
captured frame, so reference_dataset must carry it too. The
generation/reference parity check enforces this whenever the scorer declares
label_fields. Validation data must also arrive with the field, because the
strategy’s own scorer produces built-in signals only and cannot fill it in. At
least one built-in teacher_* target, or an explicit teacher_signals, is
still required alongside it. A scorer that declares neither label_fields
nor built-in signals writes fields that cannot be known before it scores a
batch. The strategy then warns that the parity check is deferred to the first
segment’s loader.
Because on_policy and reference_dataset hold live runtime objects,
to_spec_dict()
leaves them out and warns. A strategy rebuilt from that spec runs offline until
they are supplied again. Every rebuild entry point takes them as keyword
arguments:
from_spec_dict(),
from_checkpoint_dict(),
and
load_checkpoint()
all accept on_policy and reference_dataset. The checkpoint entry points
pass them to from_spec_dict as runtime overrides: live objects that a
spec cannot carry, which nvalchemi.training.load_checkpoint() forwards as
extra keyword arguments. The segment loop is tied to the student it
propagates, so pass back the models the propagator was built around as
well, through the loader’s own models= keyword. The checkpoint’s weights
are then restored into those very objects.
Alternatively, calling
restore_checkpoint() on a strategy
already constructed with the loop reaches the same result. The
re-supply is mandatory for a Boltzmann term, which matches the teacher’s
Boltzmann distribution over the configurations the student generated (see
Embedding, Hessian, and Boltzmann objectives). The term is defined only on
generated batches, so it refuses to rebuild as an offline run.
Relaxation#
A relaxation propagator generates paths that end. fmax tells the segment
loop about that and turns on the trajectory lifecycle described below.
fmax is the max-force-norm threshold, a plain number a recipe can hold.
convergence_hook instead takes a
ConvergenceHook when the run needs a live hook
rather than a number. convergence_criterion resolves the two into one
criterion. For the duration of the run, the loop installs that criterion on the
propagator in two roles: as the hook that migrates status, and as the
convergence detector. Graduation and detection therefore cannot disagree. A
detector the propagator was built with is set aside and restored afterwards.
max_trajectory_steps turns the same lifecycle on for trajectories that
never converge: a trajectory that reaches that many propagator steps is frozen,
stored once, and retired and backfilled like a converged one. It works alone,
for a thermostatted run that should rotate through its initial structures, or
beside a criterion. quarantine_diverged (default False) zeroes a
diverged graph’s velocities and forces and makes its positions finite on the
step it is flagged, so the frozen graph the propagator keeps evaluating stays a
finite model input until the boundary retires it.
A hook passed as convergence_hook must migrate status on every step, from
the status 0 the run stamps its structures with. A criterion that only
reports convergence would look configured while freezing and graduating
nothing. A criterion that skips steps would graduate a structure late, and both
capture routes (see below) would store that frame. The lifecycle must also be
the only thing that migrates status. A propagator that already carries its own
status-migrating ConvergenceHook, or its own sampler, is therefore refused;
it would otherwise run at two thresholds or refill mid-segment. A
multi-sub-stage FusedStage is refused at
construction, where that shape is fixed: the stage builds a migrator for every
sub-stage except the last, and for the last one too when it declares a
convergence_hook.
The construction probe runs the propagator’s compute() on one row. It also
dispatches a copy of the criterion to that row, stamped with the status the
run gives its structures. A criterion that raises on the propagator’s outputs,
or whose firing leaves the status column unchanged where it converged, is
refused before a run is paid for. The probe does not check whether the
structure converges, which depends on the data. It checks that the migration
works. A criterion that reads a key no compute() produces is not
dispatched, because a hook may write that key during the step. A warning names
the key instead.
The lifecycle keeps the buffer filling with informative frames. A converged
structure freezes in the propagator’s step and is stored once, as the minimum
it reached. Every later capture of the segment leaves it out instead of writing
it again. At the segment boundary it graduates: it leaves the batch, and the
optimizer’s own per-structure state follows the membership change. The initial
structures then backfill the room it freed, with at most as many structures
as graduated, within the atoms they held. The backfill goes through
draw() with
on_miss="skip", so one oversized row never starves the refills behind it.
A budgeted InitialStructures packs the
initial batch and leaves the remaining rows, in order, for the backfill. An
unbudgeted one is propagated whole, so its position starts past the last row.
The batch then narrows by one trajectory per graduation, unless
recycle=True wraps the position to the front of the rows this rank owns. A
backfilled structure is restamped with fresh bookkeeping, keeping only the
system_id the source assigned. A store of minima that an earlier relaxation
graduated therefore does not arrive frozen.
A trajectory can also end by diverging. No criterion ever accepts a NaN, so the
OnPolicyConfig.divergence predicate handles that case. By default it is
nonfinite_divergence(), which flags a
graph whose positions or forces are no longer finite. A flagged graph is frozen
at exit_status on that step and kept out of both capture routes. At the
boundary it is retired and backfilled like a converged one, and one warning per
boundary counts the diverged graphs. A custom predicate takes the live frame
and returns one boolean per graph, the same shape an
AdmissionPolicy has. It is
runtime-only, and one that returns any other shape is refused on its first
dispatch. When the last trajectory finishes and nothing is left to start
another, the loop warns once and trains its remaining steps on the frames it
has.
Frames reach the buffer by two capture routes, and each frame takes exactly
one of them. The path route is
TeacherLabelHook. The lifecycle gives
it the propagator’s exit_status, so it stores only the structures still
relaxing. It labels them inline, and it narrows the frame to them before the
teacher runs rather than after, so a mostly frozen batch costs only a small
teacher pass. A run without a lifecycle leaves the hook unnarrowed, so a
propagator that manages its own convergence keeps its final frames. The
converged route is a converged-frame hook that stores each minimum once. It
captures the frame, unlabeled, at ON_GRADUATE: the stage every
BaseDynamics and
FusedStage propagator dispatches with the graphs
whose status reached exit_status on the step. A fused stage’s own
ON_CONVERGE fires on its sub-stages only; it dispatches ON_GRADUATE
after its step-budget migration. A
DomainParallel propagator dispatches no
ON_GRADUATE, so OnPolicyConfig refuses one with a criterion rather
than let its minima go uncaptured. The hook leaves out a graph the divergence
predicate has flagged, and a graph whose final frame the path route stored on
the same step. The converged frames are labeled in a single teacher pass when
the hook’s sink is drained, which keeps the teacher’s batch size independent of
the propagated one.
The path route stages its frames in OnPolicyConfig.capture_sink when one
is configured. Every segment needs (generation_steps + 1) frames per
trajectory in the batch. A sink too small for that is grown through
resize(capacity) when it offers one, and refused up front when it does not.
The sink is never shrunk. A backfill never grows the batch past its initial
size, so a sink that fits the initial batch fits every later one, and the
growth happens at most once, for the first segment. The converged route keeps
its own host-memory sink, one frame per graph.
A custom StructureSource
drives the lifecycle too, provided its initial_batch stamps the status
zeros and system_id numbers the lifecycle graduates and backfills on.
Distribution-matching objectives are defined on equilibrium ensembles, which a
relaxation path is not. A Boltzmann term is therefore refused at construction
beside a relaxation propagator. When on_policy.samples_equilibrium is left
None, whether the propagator samples an equilibrium ensemble is inferred
from the propagator itself: from the
samples_equilibrium its class
declares, which the relaxation optimizers set to False, and from the
convergence criteria it or the loop carries. True admits a propagator that
the inference would refuse, and False refuses one that it would admit.
Pointwise energy,
force, and atomic-energy matching distill a relaxation path exactly as they
distill a trajectory.
Scaling out: multi-GPU and multi-node#
On-policy distillation scales as synchronous data parallelism. The teacher is
frozen and only runs forward passes, so a teacher that fits on one accelerator
is replicated onto every rank, while the student is trained data-parallel.
Each rank generates its own trajectories, labels them with its own teacher
replica, and fills its own replay buffer. The only per-step training traffic
between ranks is the student’s gradient all-reduce. Setup adds small
collectives, which check the shards and the replay placement, and validation
all-reduces its metrics. The teacher never joins a collective. The script is
the single-process one plus a DDPHook,
launched with one process per GPU:
strategy = DistillationStrategy(
models={"student": student, "teacher": teacher},
optimizer_configs={
"student": [OptimizerConfig(optimizer_cls=torch.optim.Adam)]
},
loss_fn=(
EnergyMSELoss(target_key="teacher_energy")
+ ForceMSELoss(target_key="teacher_forces")
),
num_steps=10_000,
devices=[torch.device("cuda")],
hooks=[
DDPHook(),
CheckpointHook("runs/distill/checkpoints", epoch_interval=1),
],
reference_dataset=labeled_store,
on_policy=OnPolicyConfig(
dynamics=propagator,
teacher_scorer=scorer,
initial_structures=InitialStructures(structure_store),
replay_ratio=0.5,
training_steps_per_segment=32,
),
)
strategy.run()
# One node, one process per GPU.
torchrun --standalone --nproc_per_node=8 distill.py
# Four nodes, run on each of them.
torchrun --nnodes=4 --nproc_per_node=8 --rdzv_backend=c10d \
--rdzv_id=distill --rdzv_endpoint=$HOST:29500 distill.py
DDPHook wraps every optimizer-configured model, so it wraps the student and
never the teacher, and it pins each rank to its node-local device. The segment
loop adds the sharding that the generation phase needs. The initial structures
are dealt out strided: rank r takes every world_size-th row, starting
at row r. They must therefore hold at least one structure per rank, and are
best sized as a whole multiple of the world size. A set that does not divide
evenly triggers a warning. Every rank draws the same number of replay samples
from a replay buffer that holds only its own trajectories, so the frames of a
shorter shard are drawn more often. The deal balances the number of rows, not
the work, so sort the dataset by atom count when structure sizes vary. The rows
a rank owns are public as
structure_shard,
and every backfill on that rank draws from them alone.
Each rank offsets the mixture sampler’s OnPolicyConfig.seed, and every
random_seed that the propagator and its sub-stages expose, by its global
rank times rank_seed_stride. The propagator side goes through
seed_offset(), which walks the fused
sub-stages and reports what it cannot move. Both streams add a step counter to
their seed, so the stride has to stay above every counter the run reaches. The
default, the prime 1_000_003, does so for a run whose counters stay below
it. Set a different stride when a replicate launch’s seeds would land on another
rank’s stride. A stage that holds randomness the offset cannot move, such as a
torch.Generator with no integer random_seed, is named in a warning
from every rank, and the caller must give it a rank-distinct seed. This matters
most when the initial structures are replicas of one geometry, because sharding
then separates nothing.
A multi-rank launch whose student nothing wraps is refused. The check is only
that something owns models["student"] after setup, so a wrapper of your
own passes it just as DDPHook does. A student handed over already wrapped
passes only as a DistributedDataParallel or FSDP
replica, the wrappers known to reduce gradients, since setup replaces nothing
for the check to compare. A wrapper that works in place, such as FSDP2’s
fully_shard or hook-based gradient synchronization, leaves nothing for the
check to find either. For such a wrapper, require_wrapped_student=False
waives the check with a one-time warning, and keeping the ranks’ students in
step becomes your responsibility.
The reference dataset is not sharded. Every rank draws from all of it with
replacement, so ranks share reference samples, while the generated frames and
the teacher passes that label them are partitioned. Each rank’s replay buffer
is staged on the reference dataset’s device. Keep that dataset in host memory,
or let it emit lazily: a Dataset
opened with no device, or with an index-less "cuda", draws its first
batch after DDPHook has pinned the rank, so the batch lands on that rank’s
GPU. Staging the dataset eagerly on a GPU before the pin concentrates the whole
world’s replay buffers on that one GPU, and every rank warns about it. Moving
the dataset onto ctx.workflow.devices[0] in a TrainingStage.SETUP hook
places it correctly. An index-less replay_device names the device this rank
has made current.
Multi-node runs take the same code path with a larger world. Sharding keys on
the global rank, and device placement on the node-local rank. The c10d
rendezvous above is what lets one command run on every node. Validation runs on
every rank and all-reduces its metrics, so never guard it behind a rank check.
CheckpointHook writes from global rank zero
only.
A restart resumes the optimizer state and the counters, reseeds every rank’s
trajectories from that rank’s shard, and refills the replay buffer from
scratch, so budget the first segments after a restart as cold. A restart needs
no device bookkeeping:
restore_checkpoint() loads onto the
live devices, and run() re-homes the optimizer state after the hook has
pinned the rank.
Every rank runs the same number of segments, and the same number of batches per
segment, which keeps the ranks arriving at each all-reduce together. An update
orchestrator that vetoes optimizer steps unevenly across ranks would
desynchronize them. A stalled rank blocks its peers for the process group’s
default timeout. DDPHook exposes no timeout setting, so to bound the wait,
initialize the process group yourself with timeout=.
The world divides the generation work. A segment’s total frame count and
teacher cost equal the single-process run’s, with each rank contributing
1/world_size of them. generation_steps, label_frequency, and
replay_capacity are per rank. At a fixed replay_capacity, each rank’s
buffer therefore spans world_size times as many segments, and every mixed
batch grows staler as the world grows. Raise generation_steps or the
structure count with the world, or lower replay_capacity by the world size,
but not both.
Losses#
Every teacher signal shaped like a total energy, a force, or a stress is
consumed by a built-in loss term with its target_key pointed at the teacher
field — EnergyMSELoss(target_key="teacher_energy"), and so on. Signals with
no supervised counterpart get their own term.
ComposedLossFunction renormalizes its weights by
default, so composed weights are relative ratios: a + b + 0.2 * c runs at
1/2.2, 1/2.2, and 0.2/2.2. Build the composition with
normalize_weights=False for literal coefficients, which also keeps a weight
schedule on one term from rescaling the others as it ramps.
Mean-squared-error loss on per-atom energies. |
Embedding, Hessian, and Boltzmann objectives#
Three further terms distill quantities that a reference dataset has no column
for. Each needs more from the run than a target field. Each is checked at
construction, in the training loss and in a validation_config loss alike.
EmbeddingMatchingLoss is the
embedding term: it matches the student’s per-atom representation (its node
embeddings) to the teacher’s, component by component. Both sides come from
compute_embeddings rather than from a forward pass, so the term needs
embedding_distillation_fn(), and the
student runs twice per batch. Across architectures the two embedding widths
differ, and the learnable
EmbeddingProjector reconciles them.
Construct it with the student’s width and the teacher’s width, and register it
as a "projector" model with an optimizer_configs entry of its own. The
training function then routes the student’s embeddings through it. The
projection is applied to the student and never to the teacher, whose
embeddings stay fixed targets. A learnable map on the target side would
minimize the objective by collapsing the teacher’s representation. The
projector is a training-time artifact: the distilled model is the student
alone.
The training function refuses student embeddings that are detached from the
student’s trainable parameters, treating them as an accident. A wrapper that
computes them under torch.no_grad produces such embeddings, and training on
them would leave the projector absorbing the term. A student representation
frozen on purpose, such as frozen trunk layers beside a trainable head, is
declared by registering the projector with frozen_student=True, so that the
projector alone carries the term.
from nvalchemi.training.distillation import (
DistillationStrategy,
EmbeddingMatchingLoss,
EmbeddingProjector,
embedding_distillation_fn,
)
projector = EmbeddingProjector(student_width, teacher_width)
strategy = DistillationStrategy(
models={"student": student, "teacher": teacher, "projector": projector},
optimizer_configs={
"student": [OptimizerConfig(optimizer_cls=torch.optim.Adam)],
"projector": [OptimizerConfig(optimizer_cls=torch.optim.Adam)],
},
loss_fn=EnergyMSELoss(target_key="teacher_energy")
+ 0.1 * EmbeddingMatchingLoss(),
training_fn=embedding_distillation_fn,
num_steps=10_000,
)
Two representations agree only up to the symmetries of each architecture’s embedding space, such as a channel permutation or a rotation of an equivariant block. Both are linear maps, so a linear projector absorbs them; without a projector they are matched component by component and leave a residual floor. Differences that no linear map closes, such as a feature one architecture builds and the other does not, leave a floor with or without a projector, which is why a residual floor on the embedding term is normal. Weight the term as a regularizer beside the terms that carry the physical targets.
HessianMatchingLoss matches the
curvature of the teacher’s energy surface. Curvature decides vibrational
spectra and integrator stability, and energies and forces do not pin it down.
Neither side forms a Hessian. Both sides compute a Hessian-vector product along
one random probe direction, at a few backward passes each. The hessian signal
materializes the teacher’s product and its probe onto the batch, either offline
through label_dataset() or on the fly
through the strategy’s labeling seam. The student’s product comes from
hessian_distillation_fn(), which takes it
on a second student pass narrowed to the energy alone. The second pass is
needed because a conservative student derives its forces from the very graph
the second derivative needs, and frees that graph outside training mode, so
the stock forward cannot be differentiated again. Every validation pass costs
the same two passes.
One probe constrains one direction, so coverage comes from redrawing the probe.
An on-policy run gets a fresh probe every time it labels a frame, whereas a
store labeled once freezes one direction per structure. Because each probe
component is standard normal, the graph-balanced value is a Hutchinson estimate
of ||dH||_F^2 / 3V in (eV/A^2)^2. For a near-converged student, that value
is one to two orders of magnitude above the force mean-squared error. Start the
term a hundred to ten thousand times lighter than the force term, and treat a
single batch’s value as the noisy one-sample estimate it is. For a direct-force
student, the strategy warns that the term supervises the energy head alone.
BoltzmannMatchingLoss implements
Boltzmann matching: it matches the distribution of configurations rather than
each configuration. The term is the relative entropy between the teacher’s and
student’s Boltzmann distributions at a given temperature. It is blind to a
constant energy offset and to any error that does not change relative
populations. beta interpolates between the forward direction (0,
mass-covering) and the reverse direction (1, mode-seeking).
The estimator reads a batch as a sample of the student’s own canonical
ensemble, which is what makes the student-side weights uniform. The strategy
therefore requires on_policy. It refuses a relaxation propagator and any
convergence criterion: the propagator’s own hook, a
ConvergenceHook registered on it that
graduates graphs out, or one the segment loop installs from fmax or
convergence_hook. It also warns when replay_ratio mixes reference
frames that the student never visited into the batch. Reweighting an
off-policy sample back onto the student’s distribution is not offered, so an
existing dataset reaches the term as reference_dataset, mixed into
generated frames by replay_ratio.
The batch also has to hold configurations of one system, because energies of
different systems are not comparable. Seed the run with replicas of one
structure, one walker per graph. The one-system check compares atom counts
only: it refuses a batch whose graphs differ in size, and it cannot tell two
systems of the same size apart. Under data parallelism, each rank holds a
shard of one world batch, the union of every rank’s batch. The check reads
the gathered world batch, so a rank holding one system beside a rank holding
another of a different size is refused too. check_one_system=False turns
the check off for an ensemble whose composition is meant to vary. A graph
whose teacher or student energy is not finite is dropped from the ensemble
(ignore_nonfinite), since one such row would otherwise reach every rank’s
softmax through the gather. The temperature cannot be checked: set the term’s
temperature and the thermostat’s from the same number.
The two directions differ in scale. The forward direction is bounded above by
log B, and its gradient vanishes once the softmax saturates, as it does for
a student whose error spreads over more than a few k_B T. beta=0 can
therefore read as converged while the student is far off. Hold beta at
0.5 or above until the student is within a couple of k_B T. Reducing
energies by k_B T also puts the gradient of either direction at up to
1/k_B T per configuration, about 39 eV^-1 at 300 K, well above what a
pointwise energy term produces. Under a
DDPHook, the term gathers the reduced
energies across ranks with a differentiable all-gather and normalizes the
softmax over the world batch. Each rank reports the world loss, the averaged
gradient is the world loss’s own, and the distribution the softmax sees is
world_size times batch_size wide.
The recommended recipe is therefore replay_ratio=1 and a bounded
replay_capacity. The ratio keeps reference rows out of the batch. The
capacity keeps stale generated rows out, because every segment’s loader draws
uniformly over the whole replay buffer and an unbounded buffer retires nothing.
Size the capacity to the frames that one segment or a few segments yield.
Validation is the other off-policy path, and the strategy refuses it outright.
A Boltzmann term in the validation loss is refused at construction, and so is
a ValidationConfig without a loss_fn of its own, which would reuse a
training loss that holds the term. Give the validation config a pointwise loss
instead.
Mean-squared-error loss on per-atom representations. |
|
Learnable map from the student's embedding width to the teacher's. |
|
Mean-squared-error loss on Hessian-vector products. |
|
Relative entropy between the teacher's and student's Boltzmann distributions. |
Evaluation and acceptance#
nvalchemi.training.distillation.evaluation answers whether a distilled
student is good enough to ship. It is its own subpackage: its names are
imported from nvalchemi.training.distillation.evaluation, not from
nvalchemi.training.distillation. A student is accepted when it clears every
acceptance bar it is given. An acceptance bar is a limit on
one number from the student’s measurements, such as a maximum force MAE or a
maximum energy drift rate.
Accuracy is measured over a held-out set with
evaluate_accuracy(). It
compares the student against either the dataset’s own labels
(targets="reference") or the teacher’s labels (targets="teacher").
Teacher labels are written offline by
label_dataset() or scored on the fly by a
scorer. The pass runs through ValidationLoop,
so eval mode, device placement, and the autograd policy an autograd-force
student needs behave as they do in training validation. No autocast is
applied, so the student predicts in its own dtype. A bare model passed as the
scorer is wrapped in an in-process scorer whose labels are cast to
label_dtype, by default the student’s first floating-point parameter dtype
floored at float32. A supplied scorer’s labels are not cast. Every residual is
accumulated in float64 as an exact global sum rather than read off the
graph-balanced loss.
The weights scored are exactly the ones passed in. To score a student trained
under an EMAHook on its averaged weights, pass
strategy.inference_model. StudentEvaluation.weights records which of
the two weight sets was scored. A scorer’s labels pass the same teacher_*
namespace guard that every other labeling route applies. A custom scorer that
returns energy or positions is therefore refused before it can rewrite
the inputs or reference targets of the batch the student is about to read.
Whenever forces are compared, against either target family, two
force-alignment numbers fill in. force_cosine_mean weights every atom
equally, so atoms whose force is at or below the student’s own error dominate
it. The min_force_cosine bar therefore reads the
magnitude-weighted force_cosine_aggregate instead. The compared quantities
form an open table. The built-ins are
BUILTIN_ACCURACY_QUANTITIES,
one AccuracyQuantitySpec
each. A spec names the prediction key, the reference field, the teacher signal,
and the loss term that drives the pass. A spec passed in quantities scores
a custom head, or a built-in quantity read from another field, and reports it
under AccuracyMetrics.errors.
Measure a student's error over a held-out set. |
|
Errors of one student against one set of targets over a held-out set. |
|
Where one accuracy quantity's prediction and target are read from. |
- nvalchemi.training.distillation.evaluation.BUILTIN_ACCURACY_QUANTITIES: Mapping[str, AccuracyQuantitySpec]#
Specs of the built-in quantities, keyed by the name a caller requests them under.
The non-conservative residual is the part of a teacher’s force field that no
conservative student can fit.
non_conservative_residual()
measures it. A student that differentiates an energy produces a curl-free
field, so it fits only the conservative part of a direct-force teacher. The
probe integrates the teacher’s work around closed loops in configuration space.
That work is zero for a conservative field. The probe converts the leftover
work into force_floor: a lower bound on the root-mean-square per-atom force
error that a conservative student must make somewhere on the loop.
The bound holds at the displacement scale that amplitude probes, so choose
an amplitude on the order of a thermal vibration. The bound loosens as
1 / sqrt(N) with system size, because one randomly oriented loop spans a
1 / sqrt(3N) fraction of the field’s curl. Compare floors only between
probes of similar system size. A conservative teacher reports the midpoint
rule’s quadrature error, which falls as segments rises. Below that, it
reports the round-off of the batch’s own dtype. In float32 that round-off is
near 1e-9 eV/A for an argon-like Lennard-Jones solid, and a hundred times
more for a lattice a hundred times stiffer. A floor below that level needs a
float64 batch and teacher.
Estimate the part of a teacher's force field no conservative student can fit. |
|
Non-conservative component of a teacher's force field. |
Stability is what small students actually fail at, so it is measured on a
trajectory the student drives itself. The stability monitor,
StabilityMonitor, is a dynamics hook that
records the total energy and momentum of that trajectory. It reports energy
drift and momentum conservation once the run is over. It is the offline
counterpart of EnergyDriftMonitorHook: it
keeps the whole series instead of comparing one live value against a
threshold. The monitor, its StabilityMetrics
record, and total_momentum() live in
nvalchemi.dynamics.hooks, because nothing about them is specific to
distillation; the evaluation suite re-exports them.
The endpoint drift and the fitted per-nanosecond rate both include whatever
transient the series starts with. A student seeded from frames that are not
equilibria of its own potential therefore needs a warmup_steps window that
covers the relaxation. Without one, the transient is reported as drift and can
cancel a genuine drift. Read energy_fluctuation_per_atom beside the rate: a
drift no larger than the fluctuation is a line through an oscillation rather
than a trend. Momentum is conserved only by an integrator that conserves it, so
set no max_momentum_drift bar under a stochastic thermostat.
The monitor’s divergence predicate defaults to
nonfinite_graph_mask(), the same predicate the
on-policy loop uses. The first firing that flags a graph ends the
series, and its step is recorded as first_divergence_step. A trajectory
that blew up is therefore scored on the segment before it did, not on
non-finite samples; set the no_divergence bar so the report fails it
rather than reading that segment’s drift. aggregate="mean" reports the
figures as the mean over
graphs instead of the worst graph. Recording stops with a warning when the
batch composition changes, so a propagator that graduates systems mid-run is
scored on the segment before the first graduation.
stop_on_composition_change=False keeps recording through an inflight refill
that preserves every graph’s size.
Extensivity is the property that a structure’s energy grows in proportion to
its size: a k-fold supercell has k times the energy of the cell it
replicates.
extensivity_error() checks
that the student’s energy scales this way across replicated cells. The
radial distribution g(r) is the pair correlation function: how often
pairs of atoms sit at separation r, normalized so an ideal gas gives 1.
radial_distribution()
accumulates it over a trajectory, reading frames straight out of a
DataSink filled by
SnapshotHook.
compare_radial_distributions()
scores the structure a trajectory samples against a reference trajectory’s
structure with a bounded Jensen-Shannon divergence. By default the comparison
pools every species into one histogram. That histogram cannot see a student
that puts the right distances between the wrong kinds of atom. Pass pair
to resolve one species pair and gate a chemically ordered system on its
partial curves. The extensivity and radial distribution checks read
periodicity from pbc when a batch carries it. Both therefore refuse a
cluster stored with a box but no periodic axis, like one without a cell.
Check that a model's energy scales with the number of replicated cells. |
|
Energy-scaling error of a model across replicated cells. |
|
Accumulate the radial distribution function of a trajectory. |
|
Radial distribution function accumulated over one or more frames. |
|
Score how far a candidate trajectory's structure sits from a reference. |
|
Scalar divergences between two radial distribution functions. |
The monitor, its record, and total_momentum are documented on the
dynamics hooks page.
The verdict is assembled from those measurements. A caller collects one
StudentEvaluation per
candidate student and sets the acceptance bars on an
AcceptanceThresholds.
build_acceptance_report()
then returns a report that renders as Rich tables and exports as a plain
dictionary or a flat scalar map. A bar with no measurement behind it fails the
student rather than being skipped. A check whose family was measured but whose
own number was not says which quantity or timestep was missing. A measurement
that is not finite fails its bar with a not finite detail, because a NaN
would fail every comparison and read as an ordinary miss, -inf would clear
every maximum, and +inf every minimum.
The divergence bar, no_divergence, is a flag rather than a limit. Set it
to True and a student whose stability series stopped on a divergence fails
its divergence row, which reports 1.0 against a limit of 0.0 and
names the step the monitor’s predicate fired at. The drift bars alone cannot
catch a divergence, because the series ends at that step and they describe the
segment before it, which can look perfectly conserved. Like every bar, it
defaults to not testing, and it fails a student with no stability record.
The from-scratch bar, max_from_scratch_ratio, compares the distilled
student against an equal-size student trained from scratch. It takes the ratio
of the distilled error to the from-scratch error on energy_per_atom_mae,
forces_mae, and stress_mae, whichever both carry, and keeps the worst
ratio. That ratio has to be at most the bar.
Both students must be scored on the same held-out set, which the bar checks. A
family scored on different held-out sets is rejected outright.
Each measurement a StudentEvaluation holds, and the evaluation itself, is a
measurement record: a frozen result, built on
MeasurementRecord, that
exports with to_dict and rebuilds from its own export with from_dict. A
sweep that evaluates each student in its own job can therefore persist the
results and assemble one report at the end. A student entry taken out of a
report export also rebuilds, with its verdict dropped. A caller that runs only
part of the suite asks
measured_bars() which bars
its measurements can decide. The answer depends on the families the caller
filled and, for accuracy, on the quantities the pass compared.
The bars themselves are a public table.
DEFAULT_BARS is the tuple of
AcceptanceBar entries the
report applies by default. Each entry names the threshold it is set under, the
families it reads, and the field it gates.
BAR_FAMILIES is derived from
it. A measurement outside the typed slots is filed under
StudentEvaluation.extra as a flat map of numbers per family. A bar whose
family is "extra:<family>" gates it, and the bar’s limit is set under the
bar’s name in AcceptanceThresholds.extra. Pass the extended table as
build_acceptance_report(..., bars=...) and measured_bars(..., bars=...).
A limit set for a bar that the table does not carry is refused rather than
skipped.
Turn a family of student evaluations into verdicts. |
|
Verdicts, the student summary, and the exports a workflow logs. |
|
Acceptance bars a student has to clear to be accepted. |
|
One threshold applied to one measurement. |
|
Everything measured about one candidate student. |
|
Outcome of every check applied to one student. |
|
Frozen measurement that exports as a plain dictionary and rebuilds from one. |
|
Return the acceptance bars families hold enough measurements to decide. |
|
Where one acceptance bar reads the number it gates, and which way it passes. |
- nvalchemi.training.distillation.evaluation.DEFAULT_BARS: tuple[AcceptanceBar, ...]#
Built-in bars of
AcceptanceThresholds, in the order they are applied.
- nvalchemi.training.distillation.evaluation.BAR_FAMILIES: Mapping[str, frozenset[str]]#
Measurement families each built-in bar reads, keyed by threshold field.