losses#
Shared speculative distillation objectives.
Functions
Compute LK lambda with a frozen teacher and one detached blend weight per call. |
- masked_lk_lambda_loss(teacher_logits, student_logits, loss_mask, *, eta=3.0)#
Compute LK lambda with a frozen teacher and one detached blend weight per call.
Implements https://arxiv.org/abs/2602.23881, sections 4.2 and 4.4. Call once per aligned draft depth. The mask averages over the local microbatch, without DP reductions.
- Parameters:
teacher_logits (Tensor) – Full-vocabulary target logits, shaped [batch, sequence, vocabulary].
student_logits (Tensor) – Aligned draft logits over the same full vocabulary as the teacher.
loss_mask (Tensor) – Nonnegative valid-token weights shaped [batch, sequence].
eta (float) – Nonnegative decay rate; zero selects pure KL.
- Returns:
FP32 masked loss. Only student logits receive gradients; an empty mask gives zero.
- Return type:
Tensor