nvalchemi.training.distillation.embedding_distillation_fn#
- nvalchemi.training.distillation.embedding_distillation_fn(models, batch)[source]#
Run the student forward pass and add its node embeddings as a prediction.
The student’s node embeddings come from
compute_embeddings(), a second pass over the batch, which is whyEmbeddingMatchingLossneeds this training function. The batch is left as it was found. When the strategy has a"projector"model, the projector is applied to the student’s embeddings before they are returned, never to the teacher’s, and it trains through its ownoptimizer_configsentry.- Parameters:
models (Mapping[str, BaseModelMixin]) – Named models of the strategy;
"student"and, when present,"projector"are read.batch (Batch) – Input batch of atomic graphs.
- Returns:
The stock
predicted_*outputs pluspredicted_node_embeddings.- Return type:
dict[str, torch.Tensor]
- Raises:
RuntimeError – If the student’s
compute_embeddingswrites nonode_embeddings. Also if, while gradients are enabled and at least one student parameter is trainable, it writes embeddings detached from those parameters, as a wrapper that computes them undertorch.no_grad()does; the embedding term could not train the student through them. A student with no trainable parameter skips this second check, since there is nothing for the embeddings to be detached from. A projector that declaresfrozen_student=Truewaives it too, for a student whose representation is frozen on purpose so that the projector alone carries the term.
See also
nvalchemi.training.distillation.EmbeddingProjectorThe width adapter.
Notes
The student runs twice per batch, which doubles its share of a training step.
compute_embeddingsis not part of the interface aDistributedDataParallelreplica proxies, so the embedding pass runs on the wrapped module. Its gradients are still reduced. However,find_unused_parameters=Truecannot see a student submodule exercised only bycompute_embeddings; that combination is the one configuration to avoid.