Adam and RMSProp on the Second Update — hard Mixed: Adam & RMSProp problem | Incognition
Adam and RMSProp on the Second Update
30 pts · 30 coins
Mixed: Adam & RMSPropHard
Adam and RMSProp on the Second Update
Two optimisers are run on the same parameter from the same start, with α=0.02 and ϵ small enough to ignore. Both receive the identical gradient gt=0.5 at every step, and both begin with all accumulators at zero.
RMSProp st=β2st−1+(1−β2)gt2,Δθt=−stαgt
Adam mt=β1mt−1+(1−β1)gt,st=β2st−1+(1−β2)gt2,Δθt=−stαmt
with β1=0.9 and β2=0.99.
On the second update (t=2), what is the size ∣Δθ2∣ of each optimiser's step?