RMSProp's Very First Step — medium Variants of Gradient Descent problem | Incognition
RMSProp's Very First Step
20 pts · 20 coins
Variants of Gradient DescentMedium
RMSProp's Very First Step
RMSProp updates a parameter with
st=ρst−1+(1−ρ)gt2,θt+1=θt−st+ϵαgt
Training has only just started, so the running average is still empty: s0=0. Take ρ=0.9, α=0.01, θ0=2, and ϵ=10−8 (small enough to ignore in the arithmetic).