Two Steps Down, With the Learning Rate Shrinking

A model has exactly one parameter, $w$ . Its loss is built in two layers — an inner score

$u(w) \;=\; w^2 - 6w + 13$

and the loss is that score squared:

$L(w) \;=\; \big(u(w)\big)^2 \;=\; \big(w^2 - 6w + 13\big)^2$

Training starts at $w_0 = 5$ and runs ordinary gradient descent,

$w_{t+1} \;=\; w_t \;-\; \eta_t \, L'(w_t)$

except that the learning rate is decayed as training goes on, following $\eta_t = \dfrac{0.02}{1+t}$ . So the two updates are not taken with the same step size:

update	moves	learning rate used
$t = 0$	$w_0 \to w_1$	$\eta_0 = 0.02$
$t = 1$	$w_1 \to w_2$	$\eta_1 = 0.01$

Run exactly two updates and report $w_2$ .

Recompute the gradient from scratch at each new point, and keep at least four decimal places in your intermediate work.

Give your answer rounded to two decimal places — for example 1.23.

Two Steps Down, With the Learning Rate Shrinking

Hints