A Longer Ramp

A team trains the same network on the same data several times, changing only the learning-rate schedule. Every run uses warmup, then cosine decay:

during warmup (steps $0$ to $W$ ) the rate climbs in a straight line: at step $t$ it is $\text{peak} \times t / W$ ;
after warmup it eases down along a cosine curve, $\text{peak} \times \tfrac{1}{2}\left(1 + \cos\!\left(\pi \cdot \dfrac{t - W}{30{,}000 - W}\right)\right)$ , reaching $0$ at step $30{,}000$ .

Here are the runs so far:

run	peak rate	warmup $W$	what happened
A	0.006	none: starts at the peak	loss exploded to NaN at step 12
B	0.006	250 steps	trained cleanly all the way to step 30,000
C	0.008	250 steps	loss fell normally, then exploded to NaN at step 213
D	0.008	1,000 steps	loss fell normally, then exploded to NaN at step 850

A teammate reads Runs A and B as proof that warmup cures these explosions, and proposes Run E: peak 0.008, with the warmup stretched to 2,000 steps.

What does the evidence predict for Run E?

A Longer Ramp

Hints