Three Runners, One Valley

Chapter 4 compares plain descent, momentum and Adam by racing them on the same landscape. Here you build that race.

The track is a stretched bowl. With weights $w_1, \dots, w_n$ and a curvature $c_i > 0$ for each coordinate,

L(w) = \tfrac{1}{2}\sum_i c_i\,w_i^2, \qquad \frac{\partial L}{\partial w_i} = c_i\,w_i

A large $c_i$ makes that direction steep and a small one makes it gentle. When the curvatures differ a lot, the bowl becomes a narrow valley.

The runners. All three start from the same point. Each takes exactly steps updates using its own learning rate $\alpha$ from rates. At every update a runner measures the gradient $g$ at its current position and then moves every coordinate at once. Everything below is per coordinate.

Plain descent ("sgd"): $w \leftarrow w - \alpha\,g$
Momentum ("momentum"), with velocity $v$ starting at $0$ :

v \leftarrow 0.9\,v + g, \qquad w \leftarrow w - \alpha\,v

Adam ("adam"), with $m$ and $s$ starting at $0$ , and update number $t = 1, 2, 3, \dots$ :

m \leftarrow 0.9\,m + 0.1\,g, \qquad s \leftarrow 0.999\,s + 0.001\,g^2

\hat{m} = \frac{m}{1 - 0.9^{\,t}}, \qquad \hat{s} = \frac{s}{1 - 0.999^{\,t}}, \qquad w \leftarrow w - \alpha\,\frac{\hat{m}}{\sqrt{\hat{s}} + 10^{-8}}

Task: write optimizer_race(curvatures, start, rates, steps).

curvatures and start are equal-length lists of floats.
rates is a dict with keys "sgd", "momentum" and "adam".

Return a dict with the same three keys, each mapped to that runner's final loss $L(w)$ , rounded to 4 decimal places. A runner whose learning rate is too large for the track still reports its loss, however huge.

Three Runners, One Valley

Examples

Hints