The Teacher's Softened Opinion

In distillation the student learns from the teacher's whole spread of probabilities, not just the right answer: cat, but dog was a close second. There's a catch at an ordinary softmax. A confident teacher puts almost all its probability on one class, and the close-second information nearly disappears. So both models' scores are first softened with a temperature $T$:

At $T = 1$ this is the ordinary softmax. A larger $T$ spreads the probabilities out more evenly.

The loss for one example blends two parts.

• Soft part. Soften the teacher's scores into $p$ and the student's scores into $q$, both at the same $T$. Measure how far $q$ is from $p$ with the KL divergence (natural log), then multiply by $T^2$:

• Hard part. Ordinary cross-entropy against the true label $y$, using the student's probabilities at temperature 1: $\ \text{hard} = -\ln q^{(1)}_y$.

The $T^2$ is there because softening shrinks the soft part's gradients by roughly a factor of $T^2$. Multiplying it back keeps the two parts on a comparable scale whatever temperature you pick.

Task: write distillation_loss(student, teacher, labels, T, alpha).

• student and teacher are lists of score lists, one per example.
• labels holds each example's correct class index.

Average the soft part over the batch, and average the hard part over the batch. Return a tuple (soft, hard, loss): the average soft part, the average hard part, and their blend. Round each to 4 decimal places, only at the end: blend the unrounded averages.