nvalchemi.training.distillation.embedding_distillation_fn#

nvalchemi.training.distillation.embedding_distillation_fn(models, batch)[source]#

Run the student forward pass and add its node embeddings as a prediction.

The student’s node embeddings come from compute_embeddings(), a second pass over the batch, which is why EmbeddingMatchingLoss needs this training function. The batch is left as it was found. When the strategy has a "projector" model, the projector is applied to the student’s embeddings before they are returned, never to the teacher’s, and it trains through its own optimizer_configs entry.

Parameters:
  • models (Mapping[str, BaseModelMixin]) – Named models of the strategy; "student" and, when present, "projector" are read.

  • batch (Batch) – Input batch of atomic graphs.

Returns:

The stock predicted_* outputs plus predicted_node_embeddings.

Return type:

dict[str, torch.Tensor]

Raises:

RuntimeError – If the student’s compute_embeddings writes no node_embeddings. Also if, while gradients are enabled and at least one student parameter is trainable, it writes embeddings detached from those parameters, as a wrapper that computes them under torch.no_grad() does; the embedding term could not train the student through them. A student with no trainable parameter skips this second check, since there is nothing for the embeddings to be detached from. A projector that declares frozen_student=True waives it too, for a student whose representation is frozen on purpose so that the projector alone carries the term.

Notes

The student runs twice per batch, which doubles its share of a training step. compute_embeddings is not part of the interface a DistributedDataParallel replica proxies, so the embedding pass runs on the wrapped module. Its gradients are still reduced. However, find_unused_parameters=True cannot see a student submodule exercised only by compute_embeddings; that combination is the one configuration to avoid.