RMSProp Rescales Every Step

A single parameter $w$ is trained on

$J(w) = (3w - 3)^2$

using RMSProp:

$g_t = \left.\frac{dJ}{dw}\right|_{w = w_{t-1}}, \qquad s_t = \beta\, s_{t-1} + (1 - \beta)\, g_t^{\,2}, \qquad w_t = w_{t-1} - \frac{\eta}{\sqrt{s_t} + \epsilon}\; g_t$

The settings are

quantity	value
starting weight $w_0$	$3$
starting accumulator $s_0$	$0$
decay $\beta$	$0.9$
learning rate $\eta$	$0.1$
$\epsilon$	$10^{-8}$ (small enough to ignore in the arithmetic)

Exactly three updates are performed. What is the value of $w_3$ ?

Round your answer to 2 decimal places. Carry at least four decimals through the intermediate steps.

RMSProp Rescales Every Step

Hints