Adam is the default optimiser for most of deep learning. It keeps two running averages per parameter — m, a smoothed gradient (momentum), and v, a smoothed squared gradient (how noisy this parameter has been) — and divides one by the square root of the other, so parameters with wild gradients take small steps and steady ones take large steps.
One update, given gradient g at timestep t:
m = β₁·m + (1 - β₁)·g | new momentum |
v = β₂·v + (1 - β₂)·g² | new second moment |
m̂ = m / (1 - β₁ᵗ) | bias-corrected |
v̂ = v / (1 - β₂ᵗ) | bias-corrected |
w = w - lr · m̂ / (√v̂ + ε) | the step |
The bias correction exists because m and v both start at zero, which drags the first few updates toward zero. Dividing by 1 - βᵗ cancels that — and it fades away as t grows.
Task: write adam_step(w, g, m, v, t, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8), where w, g, m and v are equal-length lists. Return (w_new, m_new, v_new) — three lists, each value rounded to 6 decimal places.
t starts at 1, not 0.m and v before using them, and return the uncorrected m and v — bias correction is used for the step only, never stored.