On-Policy Knowledge Distillation#

Offline distillation (see Offline Knowledge Distillation) trains a student on whatever structures the dataset happens to hold. On-policy distillation trains it on the structures the student itself visits. The student’s own propagator generates frames, the frozen teacher labels them, and the labeled frames accumulate in a replay buffer. Every training batch is a mixture of replay-buffer frames and samples from a reference dataset, a fixed teacher-labeled dataset.

Setting the strategy’s on_policy field turns run() into a loop of segments. Each segment generates frames, labels them, and trains on the mixture. The caller passes no dataloader, because each segment builds its own.

replay_ratio is the central setting. It is the fraction of every training batch drawn from the replay buffer: 1.0 trains on generated frames alone, while a lower value keeps part of each batch on the reference dataset. The reference dataset has to be teacher-labeled and carry the same fields a generated frame does, which is what label_dataset() produces below.

The teacher in this example predicts forces from its own head rather than as the negative gradient of its energy, while the student’s forces are that gradient. Distilling a non-conservative teacher into a conservative student is a supported path. Every teacher signal is detached before the student sees it, so how a force was produced never reaches the objective.

Everything runs in a few seconds with fixed seeds. DEVICE below picks the device; on CPU the printed numbers are exactly reproducible.

from __future__ import annotations

import tempfile
from collections import OrderedDict
from pathlib import Path
from typing import Any

import torch

from nvalchemi.data import AtomicData, Batch
from nvalchemi.data.datapipes import AtomicDataZarrReader, Dataset, InMemoryDataset
from nvalchemi.dynamics.integrators.nvt_langevin import NVTLangevin
from nvalchemi.hooks import TrainContext
from nvalchemi.models.base import BaseModelMixin, ModelConfig
from nvalchemi.models.demo import DemoModel, DemoModelWrapper
from nvalchemi.training import (
    EnergyMSELoss,
    ForceMSELoss,
    OptimizerConfig,
    TrainingStage,
)
from nvalchemi.training.distillation import (
    DistillationStrategy,
    InitialStructures,
    InProcessTeacherScorer,
    OnPolicyConfig,
    label_dataset,
)

Configure the run#

The run is three segments long. Each segment propagates GENERATION_STEPS steps and then takes TRAINING_STEPS_PER_SEGMENT optimizer steps on the mixture, until NUM_STEPS is reached. DEVICE picks a GPU when one is available and falls back to the CPU otherwise; set it explicitly to pin the run to one device.

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
NUM_INITIAL = 4
NUM_REFERENCE = 8
NUM_ATOMS = 4
HIDDEN_DIM = 8
NUM_STEPS = 12
TRAINING_STEPS_PER_SEGMENT = 4
GENERATION_STEPS = 5
LABEL_FREQUENCY = 1
BATCH_SIZE = 4
REPLAY_RATIO = 0.5
REPLAY_CAPACITY = 256
LEARNING_RATE = 1.0e-2
TEACHER_SEED = 11
STUDENT_SEED = 22
INITIAL_ELEMENT = 1
REFERENCE_ELEMENT = 6
SIGNALS = ["energy", "forces"]

A direct-force teacher#

The teacher’s model_config lists energy and forces in outputs. Its empty autograd_outputs says the forces come from a head, not from an energy gradient. InProcessTeacherScorer reads autograd_outputs to decide whether scoring runs under torch.enable_grad() or torch.no_grad(). An empty set is right for this teacher. A conservative teacher has to declare forces there, or its gradient is never computed. The distillation pipeline does not restrict or gate on force conservativeness: every teacher signal is detached, so the teacher stays out of the student’s autograd graph either way.

class DirectForceTeacher(torch.nn.Module, BaseModelMixin):
    """Toy potential whose forces come from a head, not from an energy gradient."""

    def __init__(self, *, hidden_dim: int, seed: int) -> None:
        super().__init__()
        torch.manual_seed(seed)
        # Checkpoint spec generation reads constructor arguments off same-named
        # attributes, so a teacher that drops them cannot be checkpointed.
        self.hidden_dim = hidden_dim
        self.seed = seed
        self.embedding = torch.nn.Embedding(16, hidden_dim)
        self.trunk = torch.nn.Sequential(
            torch.nn.Linear(hidden_dim + 3, hidden_dim),
            torch.nn.SiLU(),
        )
        self.energy_head = torch.nn.Linear(hidden_dim, 1)
        self.force_head = torch.nn.Linear(hidden_dim, 3)
        self.model_config = ModelConfig(
            outputs=frozenset({"energy", "forces"}),
            autograd_outputs=frozenset(),
            autograd_inputs=frozenset(),
            neighbor_config=None,
        )

    @property
    def embedding_shapes(self) -> dict[str, tuple[int, ...]]:
        """Return no named embeddings for this toy potential."""
        return {}

    def compute_embeddings(
        self, data: AtomicData | Batch, **kwargs: Any
    ) -> AtomicData | Batch:
        """Return ``data`` unchanged because the toy potential has no embeddings."""
        return data

    def forward(self, data: AtomicData | Batch, **kwargs: Any) -> OrderedDict:
        """Predict a total energy and per-atom forces in one pass."""
        features = self.trunk(
            torch.cat([self.embedding(data.atomic_numbers), data.positions], dim=-1)
        )
        atomic_energies = self.energy_head(features)
        batch_idx = data.batch_idx if isinstance(data, Batch) else None
        if batch_idx is None:
            energy = atomic_energies.sum(dim=0, keepdim=True)
        else:
            energy = torch.zeros(
                (data.num_graphs, 1),
                dtype=atomic_energies.dtype,
                device=atomic_energies.device,
            ).scatter_add_(0, batch_idx.unsqueeze(-1), atomic_energies)
        return self.adapt_output(
            {"energy": energy, "forces": self.force_head(features)}, data
        )


teacher = DirectForceTeacher(hidden_dim=HIDDEN_DIM, seed=TEACHER_SEED).to(DEVICE)
torch.manual_seed(STUDENT_SEED)
student = DemoModelWrapper(DemoModel(num_atom_types=16, hidden_dim=HIDDEN_DIM)).to(
    DEVICE
)
print("Teacher autograd outputs:", sorted(teacher.model_config.autograd_outputs))
print("Student autograd outputs:", sorted(student.model_config.autograd_outputs))
Teacher autograd outputs: []
Student autograd outputs: ['forces']

Initial structures and a teacher-labeled reference dataset#

Build two sets of structures, tagged by atomic number so the mixture is easy to read later. The initial structures are the bare geometries the trajectories start from; the source they go behind is covered under initial structures. The reference dataset is labeled by the same teacher, which is what makes it admissible and keeps the dtypes in step; see The reference dataset must be teacher-labeled.

def build_systems(element: int, num_systems: int, seed: int) -> Batch:
    """Return deterministic random systems tagged by *element*, geometry only."""
    generator = torch.Generator().manual_seed(seed)
    return Batch.from_data_list(
        [
            AtomicData(
                positions=torch.randn(NUM_ATOMS, 3, generator=generator),
                atomic_numbers=torch.full((NUM_ATOMS,), element, dtype=torch.long),
                atomic_masses=torch.ones(NUM_ATOMS),
            )
            for _ in range(num_systems)
        ]
    )


initial_structures = InitialStructures(
    InMemoryDataset(
        in_memory_batch=build_systems(INITIAL_ELEMENT, NUM_INITIAL, 500).to(DEVICE)
    )
)
scorer = InProcessTeacherScorer(teacher, SIGNALS)

store = Path(tempfile.mkdtemp(suffix="_on_policy")) / "reference.zarr"
label_dataset(
    InMemoryDataset(
        in_memory_batch=build_systems(REFERENCE_ELEMENT, NUM_REFERENCE, 700)
    ),
    scorer,
    store,
    batch_size=4,
    device=DEVICE,
)
reference_dataset = Dataset(reader=AtomicDataZarrReader(store), device=DEVICE)
print(
    f"Initial structures: {len(initial_structures)}, "
    f"reference structures: {len(reference_dataset)}"
)
print("Reference fields:", ", ".join(sorted(reference_dataset.field_names)))
Initial structures: 4, reference structures: 8
Reference fields: atom_categories, atomic_masses, atomic_numbers, positions, teacher_energy, teacher_forces, velocities

Configure the segment loop#

The propagator holds the very module the optimizer updates, which is what makes the generated data on-policy; the strategy checks that identity at construction. label_frequency is the throughput setting, since the teacher is the expensive model; see the cadence.

on_policy = OnPolicyConfig(
    dynamics=NVTLangevin(
        student, dt=0.5, temperature=300.0, friction=0.01, random_seed=7
    ),
    teacher_scorer=scorer,
    initial_structures=initial_structures,
    replay_ratio=REPLAY_RATIO,
    training_steps_per_segment=TRAINING_STEPS_PER_SEGMENT,
    batch_size=BATCH_SIZE,
    generation_steps=GENERATION_STEPS,
    label_frequency=LABEL_FREQUENCY,
    replay_capacity=REPLAY_CAPACITY,
)

Run the loop#

The objective reads teacher fields only, because that is all a generated frame carries. The run is sized in optimizer steps rather than epochs: every segment builds its own loader, so there is no fixed epoch to convert.

loss_fn = EnergyMSELoss(target_key="teacher_energy") + ForceMSELoss(
    target_key="teacher_forces", normalize_by_atom_count=True
)


class MixtureTrace:
    """Record the loss and the source composition of every training batch."""

    frequency = 1
    stage = TrainingStage.AFTER_BATCH

    def __init__(self) -> None:
        self.losses: list[float] = []
        self.compositions: list[tuple[int, int]] = []

    def __call__(self, ctx: TrainContext, stage: TrainingStage) -> None:
        """Append the loss and the (generated, reference) counts of this batch."""
        self.losses.append(float(ctx.loss))
        tags = [
            int(ctx.batch.atomic_numbers[ctx.batch.batch_idx == index][0])
            for index in range(ctx.batch.num_graphs)
        ]
        self.compositions.append(
            (tags.count(INITIAL_ELEMENT), tags.count(REFERENCE_ELEMENT))
        )


trace = MixtureTrace()
strategy = DistillationStrategy(
    models={"student": student, "teacher": teacher},
    optimizer_configs={
        "student": [
            OptimizerConfig(
                optimizer_cls=torch.optim.Adam,
                optimizer_kwargs={"lr": LEARNING_RATE},
            )
        ]
    },
    loss_fn=loss_fn,
    num_steps=NUM_STEPS,
    on_policy=on_policy,
    reference_dataset=reference_dataset,
    devices=[DEVICE],
    hooks=[trace],
)
print("Teacher signals:", ", ".join(sorted(strategy.teacher_scorer.signals)))

strategy.run()
Teacher signals: energy, forces

Inspect the replay buffer#

The buffer outlives the run: a second run() with a raised NUM_STEPS appends to these frames. Its schema is the contract the reference dataset had to meet: the structure plus the teacher_* labels, never the propagator’s own energy and forces. See The reference dataset must be teacher-labeled.

buffer = strategy.replay_buffer
print(f"Replay buffer: {len(buffer)} frames")
print("Buffer schema:", ", ".join(sorted(buffer.schema)))
Replay buffer: 60 frames
Buffer schema: node.atom_categories, node.atomic_masses, node.atomic_numbers, node.positions, node.teacher_forces, node.velocities, system.teacher_energy

Read the result#

Every batch has exactly the composition replay_ratio sets, not just on average. The loss is compared segment to segment, because each segment trains on a buffer the previous one grew.

first = sum(trace.losses[:TRAINING_STEPS_PER_SEGMENT]) / TRAINING_STEPS_PER_SEGMENT
last = sum(trace.losses[-TRAINING_STEPS_PER_SEGMENT:]) / TRAINING_STEPS_PER_SEGMENT
print("Batch compositions (generated, reference):", sorted(set(trace.compositions)))
print(f"Completed {strategy.step_count} steps over {strategy.epoch_count} segments")
print(f"Mean loss, first segment: {first:.4f}")
print(f"Mean loss, last segment:  {last:.4f}")
print(f"Reduction: {100.0 * (1.0 - last / first):.1f}%")
Batch compositions (generated, reference): [(2, 2)]
Completed 12 steps over 3 segments
Mean loss, first segment: 0.2493
Mean loss, last segment:  0.0675
Reduction: 72.9%

Total running time of the script: (0 minutes 0.772 seconds)

Gallery generated by Sphinx-Gallery