How Deep Before the Gradient Dies

Backpropagation carries a gradient backwards through a network, and crossing one layer multiplies it by that layer's factor

$w \cdot \sigma'(z)$

Consider a deep sigmoid network in the most favourable case possible: every layer has weight $w = 2$, and at every layer the sigmoid derivative sits at its largest achievable value, $\sigma'(z) = 0.25$.

The gradient arriving from the output has magnitude $1$. A parameter whose gradient magnitude has fallen below $10^{-3}$ makes no visible progress during training.

Passing back through how many such layers does the gradient magnitude first drop below $10^{-3}$?