nvalchemi.training.distillation.EmbeddingMatchingLoss#

class nvalchemi.training.distillation.EmbeddingMatchingLoss(*, target_key='teacher_node_embeddings', prediction_key='predicted_node_embeddings', normalize_by_atom_count=True, ignore_nonfinite=True, dtype_policy='strict')[source]#

Mean-squared-error loss on per-atom representations.

The prediction is the student’s node embeddings. The target is the teacher’s embeddings signal, which InProcessTeacherScorer writes to teacher_node_embeddings. Both are node-level tensors of shape (V, H). The per-component residual \(\rho_{iah} = (\hat{z}_{iah} - z_{iah})^2\) is reduced according to normalize_by_atom_count. By default, with \(\mathcal{V}_i\) the atoms of graph \(i\) that mask accepts and \(M_i = |\mathcal{V}_i|\), the loss is

\[L = \frac{1}{B} \sum_{i=1}^{B} \frac{1}{\max(H M_i, 1)} \sum_{a \in \mathcal{V}_i} \sum_{h=1}^{H} \rho_{iah}\]

so every structure contributes equally regardless of size. When normalize_by_atom_count is False, the loss is one global mean over every valid component.

Parameters:
  • target_key (str, default "teacher_node_embeddings") – Target container key for the teacher’s node embeddings.

  • prediction_key (str, default "predicted_node_embeddings") – Prediction container key for the student’s node embeddings. The stock student forward pass does not produce them; see the Notes.

  • normalize_by_atom_count (bool, default True) – When True, compute a mean residual per graph, then mean over graphs. When False, compute one global mean over valid components.

  • ignore_nonfinite (bool, default True) – When True, components whose target is NaN or infinite are excluded from both loss value and gradient.

  • dtype_policy ({"strict", "prediction_to_target", "target_to_prediction"}, default "strict") – How to handle prediction/target dtype mismatches before validation.

Raises:

ValueError – If the student’s and teacher’s embedding widths differ, or if the graph-balanced reduction is requested without batch_idx and num_graphs metadata.

Examples

>>> import torch
>>> from nvalchemi.training.distillation import EmbeddingMatchingLoss
>>> loss_fn = EmbeddingMatchingLoss()
>>> pred = torch.tensor([[0.0, 2.0], [0.0, 0.0]])
>>> target = torch.zeros(2, 2)
>>> batch_idx = torch.tensor([0, 1])
>>> loss_fn(pred, target, batch_idx=batch_idx, num_graphs=2)
tensor(1.)

See also

EmbeddingProjector

Learnable width adapter for cross-architecture runs.

Notes

On both sides, embeddings come from compute_embeddings(), a second pass over the batch. The stock default_distillation_fn() therefore cannot serve this term, and DistillationStrategy refuses it at construction. Train with embedding_distillation_fn() instead.

Two architectures’ representations of one environment agree only up to the symmetries of each embedding space, such as a channel permutation or a rotation of an equivariant block. Both are linear maps, so a linear EmbeddingProjector absorbs them; without a projector they are matched component by component and leave a residual floor. Differences that no linear map closes, such as a feature one architecture builds and the other does not, leave a floor with or without one. Weight the term as a regularizer beside the terms that carry the physical targets.

compute_residual(pred, target, valid)[source]#

Return squared component residuals, zeroing invalid components.

Parameters:
  • pred (Float[Tensor, 'V H'])

  • target (Float[Tensor, 'V H'])

  • valid (Bool[Tensor, 'V H'])

Return type:

Float[Tensor, ‘V H’]

extra_repr()[source]#

Human-readable hyperparameter summary for nn.Module’s repr.

Return type:

str

mask(pred, target, ctx, **kwargs)[source]#

Return one validity flag per embedding component.

Parameters:
  • pred (Float[Tensor, 'V H'])

  • target (Float[Tensor, 'V H'])

  • ctx (ReductionContext)

  • kwargs (Any)

Return type:

Bool[Tensor, ‘V H’]

reduce(residual, valid, ctx, **kwargs)[source]#

Reduce squared component residuals to a scalar loss.

masked_mean() takes the global mean over valid entries, or the graph-balanced one when normalize_by_atom_count is set, in at least float32 either way.

Parameters:
  • residual (Float[Tensor, 'V H'])

  • valid (Bool[Tensor, 'V H'])

  • ctx (ReductionContext)

  • kwargs (Any)

Return type:

Tensor

validate(pred, target)[source]#

Check that the two representations agree in shape, width included.

Parameters:
  • pred (Float[Tensor, 'V H'])

  • target (Float[Tensor, 'V H'])

Return type:

None