losses#

Shared speculative distillation objectives.

Functions

masked_lk_lambda_loss

Compute LK lambda with a frozen teacher and one detached blend weight per call.

masked_lk_lambda_loss(teacher_logits, student_logits, loss_mask, *, eta=3.0)#

Compute LK lambda with a frozen teacher and one detached blend weight per call.

Implements https://arxiv.org/abs/2602.23881, sections 4.2 and 4.4. Call once per aligned draft depth. The mask averages over the local microbatch, without DP reductions.

Parameters:
  • teacher_logits (Tensor) – Full-vocabulary target logits, shaped [batch, sequence, vocabulary].

  • student_logits (Tensor) – Aligned draft logits over the same full vocabulary as the teacher.

  • loss_mask (Tensor) – Nonnegative valid-token weights shaped [batch, sequence].

  • eta (float) – Nonnegative decay rate; zero selects pure KL.

Returns:

FP32 masked loss. Only student logits receive gradients; an empty mask gives zero.

Return type:

Tensor