A Ladder of Learning Rates

The learning rate is the most important dial in training, and the usual way to set it is not theory but a quick experiment. Try a ladder of rates, each a fixed multiple of the last, give each one a short run, and watch three behaviours appear in order: crawling, working, then blowing up.

Run that experiment on a straight-line model $\hat y = w x + b$, fitted by mean squared error to $n$ points. Its gradients are

Task: write rate_sweep(xs, ys, start_rate, factor, max_rates, steps).

• The rates tried are start_rate, start_rate * factor, start_rate * factor**2, and so on: at most max_rates of them, in that order.
• Each rate gets its own short run from scratch: start at $w = b = 0$, then take steps steps of gradient descent using all the points at once. In every step, compute both gradients at the current $(w, b)$ before changing either one. The rate's result is the MSE after its last step.
• The starting loss is the MSE at $w = b = 0$. A rate blows up if its result is larger than the starting loss. The first rate that blows up ends the sweep: it is not recorded, and no larger rate is tried.

Return a tuple (chosen, losses):

• losses is the result of every recorded rate, in the order tried, each rounded to 4 decimal places.
• chosen is the recorded rate with the lowest result (on a tie, the smaller rate), rounded to 4 decimal places, or None if the very first rate blew up.