nvalchemi.training.distillation.EmbeddingMatchingLoss#
- class nvalchemi.training.distillation.EmbeddingMatchingLoss(*, target_key='teacher_node_embeddings', prediction_key='predicted_node_embeddings', normalize_by_atom_count=True, ignore_nonfinite=True, dtype_policy='strict')[source]#
Mean-squared-error loss on per-atom representations.
The prediction is the student’s node embeddings. The target is the teacher’s
embeddingssignal, whichInProcessTeacherScorerwrites toteacher_node_embeddings. Both are node-level tensors of shape(V, H). The per-component residual \(\rho_{iah} = (\hat{z}_{iah} - z_{iah})^2\) is reduced according tonormalize_by_atom_count. By default, with \(\mathcal{V}_i\) the atoms of graph \(i\) thatmaskaccepts and \(M_i = |\mathcal{V}_i|\), the loss is\[L = \frac{1}{B} \sum_{i=1}^{B} \frac{1}{\max(H M_i, 1)} \sum_{a \in \mathcal{V}_i} \sum_{h=1}^{H} \rho_{iah}\]so every structure contributes equally regardless of size. When
normalize_by_atom_countisFalse, the loss is one global mean over every valid component.- Parameters:
target_key (str, default "teacher_node_embeddings") – Target container key for the teacher’s node embeddings.
prediction_key (str, default "predicted_node_embeddings") – Prediction container key for the student’s node embeddings. The stock student forward pass does not produce them; see the Notes.
normalize_by_atom_count (bool, default True) – When
True, compute a mean residual per graph, then mean over graphs. WhenFalse, compute one global mean over valid components.ignore_nonfinite (bool, default True) – When
True, components whose target isNaNor infinite are excluded from both loss value and gradient.dtype_policy ({"strict", "prediction_to_target", "target_to_prediction"}, default "strict") – How to handle prediction/target dtype mismatches before validation.
- Raises:
ValueError – If the student’s and teacher’s embedding widths differ, or if the graph-balanced reduction is requested without
batch_idxandnum_graphsmetadata.
Examples
>>> import torch >>> from nvalchemi.training.distillation import EmbeddingMatchingLoss >>> loss_fn = EmbeddingMatchingLoss() >>> pred = torch.tensor([[0.0, 2.0], [0.0, 0.0]]) >>> target = torch.zeros(2, 2) >>> batch_idx = torch.tensor([0, 1]) >>> loss_fn(pred, target, batch_idx=batch_idx, num_graphs=2) tensor(1.)
See also
EmbeddingProjectorLearnable width adapter for cross-architecture runs.
Notes
On both sides, embeddings come from
compute_embeddings(), a second pass over the batch. The stockdefault_distillation_fn()therefore cannot serve this term, andDistillationStrategyrefuses it at construction. Train withembedding_distillation_fn()instead.Two architectures’ representations of one environment agree only up to the symmetries of each embedding space, such as a channel permutation or a rotation of an equivariant block. Both are linear maps, so a linear
EmbeddingProjectorabsorbs them; without a projector they are matched component by component and leave a residual floor. Differences that no linear map closes, such as a feature one architecture builds and the other does not, leave a floor with or without one. Weight the term as a regularizer beside the terms that carry the physical targets.- compute_residual(pred, target, valid)[source]#
Return squared component residuals, zeroing invalid components.
- Parameters:
pred (Float[Tensor, 'V H'])
target (Float[Tensor, 'V H'])
valid (Bool[Tensor, 'V H'])
- Return type:
Float[Tensor, ‘V H’]
- mask(pred, target, ctx, **kwargs)[source]#
Return one validity flag per embedding component.
- Parameters:
pred (Float[Tensor, 'V H'])
target (Float[Tensor, 'V H'])
ctx (ReductionContext)
kwargs (Any)
- Return type:
Bool[Tensor, ‘V H’]
- reduce(residual, valid, ctx, **kwargs)[source]#
Reduce squared component residuals to a scalar loss.
masked_mean()takes the global mean over valid entries, or the graph-balanced one whennormalize_by_atom_countis set, in at least float32 either way.- Parameters:
residual (Float[Tensor, 'V H'])
valid (Bool[Tensor, 'V H'])
ctx (ReductionContext)
kwargs (Any)
- Return type: