A single parameter w w w
J ( w ) = ( w 2 − 3 ) 2 J(w) = \left(w^2 - 3\right)^2 J ( w ) = ( w 2 − 3 ) 2
using Adam with bias correction . At step t t t g t g_t g t J J J w t − 1 w_{t-1} w t − 1
m t = β 1 m t − 1 + ( 1 − β 1 ) g t , v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2 m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^{\,2} m t = β 1 m t − 1 + ( 1 − β 1 ) g t , v t = β 2 v t − 1 + ( 1 − β 2 ) g t 2
m ^ t = m t 1 − β 1 t , v ^ t = v t 1 − β 2 t , w t = w t − 1 − η m ^ t v ^ t + ϵ \hat{m}_t = \frac{m_t}{1 - \beta_1^{\,t}}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^{\,t}}, \qquad w_t = w_{t-1} - \frac{\eta\,\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} m ^ t = 1 − β 1 t m t , v ^ t = 1 − β 2 t v t , w t = w t − 1 − v ^ t + ϵ η m ^ t
The settings are
quantity value starting weight w 0 w_0 w 0 2 2 2 m 0 m_0 m 0 v 0 v_0 v 0 0 0 0 0 0 0 β 1 \beta_1 β 1 0.9 0.9 0.9 β 2 \beta_2 β 2 0.999 0.999 0.999 learning rate η \eta η 0.5 0.5 0.5 ϵ \epsilon ϵ 10 − 8 10^{-8} 1 0 − 8
Exactly three updates are performed. What is the value of w 3 w_3 w 3
Round your answer to 2 decimal places . Carry at least four decimals through the intermediate steps, and note that the bias-correction divisors change at every step.