The Same Activation, Two Different Outputs — hard Statistics for Deep Learning problem | Incognition
The Same Activation, Two Different Outputs
30 pts · 30 coins
Statistics for Deep LearningHard
The Same Activation, Two Different Outputs
One neuron in a deep network is followed by a Batch Normalization layer whose learned parameters are
γ=3.0,β=−0.5
Entering this training step, the layer is carrying the running estimates
μrun=6.0,σrun2=13.0
1. The training step. A mini-batch of m=4 examples reaches this neuron with pre-normalization activations
x=[1.0,7.0,9.0,15.0]
In training mode the layer standardizes with the mini-batch's own statistics:
μB=m1∑i=1mxi,σB2=m1∑i=1m(xi−μB)2
x^=σB2+ϵx−μB,y=γx^+β
Take ϵ=0 throughout.
2. The running-statistics update. Once the batch statistics are computed, the layer refreshes both running estimates with momentum α=0.25:
μrun←(1−α)μrun+αμB,σrun2←(1−α)σrun2+ασB2
3. The evaluation step. Training then stops and the model is switched to evaluation mode. An input arrives that produces the same pre-normalization activation at this neuron, x=15.0. In evaluation mode the layer substitutes its running estimates for the batch statistics; γ and β are unchanged.
Let ytrain be this layer's output for x=15.0 during the training step, and yeval its output for x=15.0 in evaluation mode.