Two Students, One Cat

A large teacher model has been trained, and a small student is now being trained to reproduce its outputs. For one training photo, which really is a cat, the teacher outputs:

	cat	dog	car
teacher	$0.70$	$0.25$	$0.05$

Two students, partway through training, output this on the same photo:

	cat	dog	car
student P	$0.60$	$0.30$	$0.10$
student Q	$0.60$	$0.10$	$0.30$

Each student is graded with categorical cross-entropy, written for a general target $t$ :

$L = -\sum_k t_k \ln s_k$

where $s_k$ is the student's probability for class $k$ . Two targets are on offer:

Hard label: $t = (1, 0, 0)$ , cat and nothing else. With this target the formula is the familiar $-\ln(\text{probability on the true class})$ .
Soft targets: $t = (0.70, 0.25, 0.05)$ , the teacher's output.

Write $\Delta = L_Q - L_P$ for how much worse student Q scores than student P under a given target.

How much larger is $\Delta$ under soft targets than under the hard label?

Use natural logarithms and round to 2 decimal places.

Two Students, One Cat

Hints