A 3-layer network with one unit per layer and a smooth activation ϕ:
a0=x,zl=wlal−1,al=ϕ(zl)
On one batch, with input x=2, the forward pass recorded these values:
| layer l | weight wl | activation al | ϕ′(zl) |
|---|
| 1 | w1 | 0.6 | 0.40 |
| 2 | 2.5 | 0.8 | 0.36 |
| 3 | 2.0 | 0.5 | 0.75 |
and the loss contributes
∂a3∂L=2
Compute the gradient at the first layer's weight, ∂L/∂w1, and compare its size with the gradient at the last layer's weight, ∂L/∂w3.
Which pair is correct?