nvalchemi.training.distillation.EmbeddingProjector#
- class nvalchemi.training.distillation.EmbeddingProjector(in_features, out_features, *, hidden_features=None, bias=True, frozen_student=False)[source]#
Learnable map from the student’s embedding width to the teacher’s.
Embedding matching compares the student’s and teacher’s per-atom representations component by component, so the two widths must agree. Different architectures rarely share a width, and the student’s width is a capacity decision. The projector reconciles the widths with a small map on the student side, trained jointly against fixed teacher targets. It is never applied to the teacher: a learnable map on the target side minimizes the objective by collapsing the teacher’s representation.
The projector is registered as an ordinary named model of
DistillationStrategy, because named models are the only placesetup_optimizerssees a module’s parameters. It therefore needs anoptimizer_configsentry and is checkpointed like every other model. It is discarded at the end of training, because the distilled artifact is the student alone. Every constructor argument is stored as an attribute of the same name, from which a checkpoint’s model spec rebuilds the projector. The projector is an adapter rather than a model of a physical system: it declares no outputs and needs no neighbor list.forward()maps an embedding tensor, andcompute_embeddings()replaces the embeddings on a batch in place.- Parameters:
in_features (int) – Width of the student’s node embeddings.
out_features (int) – Width of the teacher’s node embeddings.
hidden_features (int | None, optional) – Width of one hidden layer, making the projector a two-layer perceptron with a
SiLUnonlinearity. DefaultNone, a single linear map.bias (bool, optional) – Whether the linear layers carry a bias. Default
True.frozen_student (bool, optional) – Whether the student’s representation is frozen on purpose, so that the embedding term (the
EmbeddingMatchingLosscomponent of the loss) trains this projector alone. WhenTrue,embedding_distillation_fn()accepts student embeddings that are detached from the student’s trainable parameters, such as those of a frozen trunk beside a trainable head. WhenFalse, it treats them as an accident and refuses them. DefaultFalse.
- Raises:
ValueError – If a width is not positive.
Examples
>>> import torch >>> from nvalchemi.training.distillation import EmbeddingProjector >>> projector = EmbeddingProjector(64, 256) >>> projector(torch.zeros(10, 64)).shape torch.Size([10, 256])
Notes
A linear projector is the default because it is the weakest map that can reconcile the widths. A projector with enough capacity to fit the teacher’s representation from any student representation satisfies the loss without the student learning anything. Set
hidden_featuresonly when the embedding term stalls under a linear map while the energy and force terms converge.- compute_embeddings(data, **kwargs)[source]#
Replace the node embeddings on data with their projection, in place.
- Parameters:
data (AtomicData | Batch) – Data already carrying node embeddings, as the student’s own
compute_embeddings()leaves it.**kwargs (Any) – Unused; accepted for interface compatibility.
- Returns:
data, with projected embeddings.
- Return type:
- Raises:
KeyError – If data carries no
node_embeddingsto project.
- property embedding_shapes: dict[str, tuple[int, ...]]#
Return the per-node shape this projector maps embeddings to.