One Temperature for the Batch

The softmax lesson mentioned an optional dial called temperature. Every logit is divided by the same number $T > 0$ before softmax:

$p_j = \frac{e^{z_j / T}}{\sum_k e^{z_k / T}}$

$T = 1$ is plain softmax. A smaller $T$ sharpens the probabilities toward the top class, and a larger $T$ flattens them toward an even split. Dividing every logit by the same positive number never changes which one is largest, so the model's answers are identical at every temperature. Only its confidence moves.

A finished classifier has scored four photos. Each row holds its logits for three animals:

The loss is categorical cross-entropy. The loss on one photo is $-\ln p_{\text{true}}$, where $p_{\text{true}}$ is the probability given to its true class and $\ln$ is the natural logarithm. The batch loss is the average over the photos.

Task: write best_temperature(batch_logits, true_classes), returning [temperature, loss_at_one, loss_at_best], each rounded to 4 decimal places:

• temperature is the value of $T$ that gives the lowest batch loss;
• loss_at_one is the batch loss at $T = 1$;
• loss_at_best is the batch loss at that best temperature.

Details:

• batch_logits[i] is the list of logits for photo i, one per class, and true_classes[i] is the index of its true class. Every photo has the same number of classes, at least $2$.
• Near $T = 0$ each photo becomes certain of its top class, so any photo whose true class is not on top costs without limit. Far above $1$ every photo drifts toward an even split. Every test batch is built so that, as $T$ sweeps upward from near $0$, the batch loss falls to a single lowest point and then rises again.
• Answers must be exact to 4 decimal places, which a grid of temperatures will not reliably reach.

The four photos above are the first test, with true_classes equal to [0, 0, 1, 2].

A best temperature above 1 means the model is overconfident: its mistakes are held too firmly, and the batch pays less once every photo is made more modest. A best temperature below 1 means it is timid about answers it mostly gets right. The answers never change either way, which is why accuracy cannot see any of this, and why cross-entropy, which grades confidence, can.