nvalchemi.training.distillation.evaluation.evaluate_accuracy#

nvalchemi.training.distillation.evaluation.evaluate_accuracy(model, data, *, targets='reference', quantities=None, scorer=None, target_keys=None, loss_fn=None, validation_fn=default_training_fn, grad_mode='auto', label_dtype=None, device=None, distributed_manager=None, name='accuracy')[source]#

Measure a student’s error over a held-out set.

The pass runs through ValidationLoop, so eval mode and device placement behave as in training validation. The autograd policy is settled before the loop runs, because a student that differentiates inside its own forward needs gradients whatever is scored. The loop is built without an autocast context, so the student predicts in its own dtype. A bare scorer model labels with autocast disabled, the in-process scorer’s default; a supplied scorer decides its own precision. The metrics are exact global residual sums accumulated in float64. The loop’s graph-balanced loss value is discarded.

targets="reference" compares against the dataset’s own labels. targets="teacher" compares against the teacher_* fields, which label_dataset() writes offline or a scorer writes on the fly. Against either family, the force-alignment numbers fill in whenever forces are compared, and the per-atom energy diagnostic whenever "atomic_energies" is requested and both sides carry it. A scorer passed with reference targets is refused, because its teacher pass would be paid for and thrown away.

Parameters:
  • model (BaseModelMixin) – Student to evaluate. The call leaves it in the training mode it arrived in, and scores exactly the weights passed. For a student trained under an EMAHook, pass strategy.inference_model. Nothing downstream can tell which weights were scored, so record the choice on StudentEvaluation.

  • data (Iterable[Batch]) – Re-iterable held-out set. One-shot iterators are rejected.

  • targets ({"reference", "teacher"}, optional) – Family of batch fields to compare against. Default "reference".

  • quantities (Sequence[str | AccuracyQuantitySpec] | None, optional) – Quantities to evaluate. Each entry is a built-in name from BUILTIN_ACCURACY_QUANTITIES, or an AccuracyQuantitySpec instance for a custom quantity, which is reported under errors. The built-in "atomic_energies" is a diagnostic that never enters the pass’s loss. Default None (energy and forces).

  • scorer (TeacherScorer | BaseModelMixin | None, optional) – Teacher that labels each batch before it is evaluated. A bare model is wrapped in an InProcessTeacherScorer for the requested quantities, and its labels are cast to label_dtype. A supplied scorer’s labels are not cast. Default None.

  • target_keys (Mapping[str, str] | None, optional) – Per-quantity overrides of the batch field to compare against. They take precedence over the fields targets selects. Default None.

  • loss_fn (ComposedLossFunction | None, optional) – Loss driving the pass. Default None (mean-squared terms over the requested supervised quantities).

  • validation_fn (Callable, optional) – Forward callable invoked as validation_fn(model, batch). Default default_training_fn().

  • grad_mode ({"auto", "enabled", "disabled"}, optional) – Autograd policy. "auto" enables gradients whenever the student’s forward needs them or the loss does. "disabled" is refused for a student whose forward differentiates. Default "auto".

  • label_dtype (torch.dtype | None, optional) – Dtype that a bare scorer model’s labels are cast to. Default None (the dtype the distillation strategy infers for the student’s own labels: the dtype of its first floating-point parameter, floored at float32).

  • device (torch.device | str | None, optional) – Device the pass runs on. Default None (the model’s own device).

  • distributed_manager (Any | None, optional) – Manager used to all-reduce the metric sums. Default None.

  • name (str, optional) – Label stored on the result. Default "accuracy".

Returns:

Errors and consistency diagnostics over the whole set.

Return type:

AccuracyMetrics

Raises:

ValueError – If quantities names an unknown quantity, repeats a name, or includes no supervised quantity. If a quantity without a teacher signal is compared against the teacher and has no target_keys entry. If a scorer is given but no requested quantity is compared against a teacher field, or the scorer does not publish the fields the evaluation reads, or it returns a label outside teacher_*. If gradients are disabled for a student that differentiates inside its forward. If a prediction and its target disagree on shape, or if no metric could be measured on any rank. If any rank received no batches: the refusal is raised on every rank, naming the empty ones, before the pass starts.

Examples

>>> from nvalchemi.training.distillation.evaluation import evaluate_accuracy
>>> metrics = evaluate_accuracy(student, holdout)
>>> metrics.forces_mae
0.031

Against the teacher, labeling on the fly:

>>> metrics = evaluate_accuracy(
...     student,
...     holdout,
...     targets="teacher",
...     scorer=teacher,
...     quantities=("energy", "forces", "atomic_energies"),
... )

Notes

The student is called through validation_fn exactly as a training loop would call it. A student that reads a neighbor list therefore needs batches that carry one. A scorer builds the teacher’s own list for each batch and rolls it back afterwards.

grad_mode="auto" reads the student’s autograd_outputs, the same declaration that adapt_input() reads, and looks through any DistributedDataParallel or similar wrapper to find it. An autograd-force student is therefore scored with gradients even when only energies are compared. Narrowing its active_outputs puts it back on the torch.no_grad() path.

Under a distributed run, every rank must call this function with the same quantities and a non-empty shard. The sums are packed in one key order before the all-reduce, so packs of different shapes would deadlock. Whether each rank has a batch is exchanged before the pass, because the validation loop raises on an empty shard before its own summary reduce and would strand the other ranks there; the refusal names the empty ranks on every rank instead. Shard sizes, and the targets each shard carries, may differ: a shard with no target for a quantity contributes zeros to its sums.