What Reaches the Last Layer

The chapter's rule for initialization is scale the spread by the number of inputs: more inputs means smaller weights, so the signal keeps roughly its size through the whole network. In this problem you check that claim one layer at a time, for two different scaling rules.

Take a stack of fully connected layers with ReLU after every layer. layer_sizes $= [n_0, n_1, \dots, n_L]$: there are $n_0$ input features, and layer $\ell$ (for $\ell = 1, \dots, L$) has $n_\ell$ neurons, each receiving all $n_{\ell-1}$ values from the layer below. Biases start at zero.

Every weight in layer $\ell$ is drawn independently at random with mean $0$ and spread (standard deviation) $\sigma_\ell$, set by scheme:

How the signal's size changes. A neuron in layer $\ell$ computes $z = w_1 a_1 + w_2 a_2 + \dots$ over its $n_{\ell-1}$ inputs. Treat the terms as independent. Each weight has mean $0$, so

$\text{Var}(z) = n_{\ell-1}\, \sigma_\ell^{2}\, m_{\ell-1}$

where $m_{\ell-1}$ is the mean square of the values arriving at layer $\ell$, meaning the average of $a^2$:

• The raw input features have mean $0$ and spread $s$ (input_spread), so $m_0 = s^2$.
• A layer's $z$ is symmetric around $0$, and its ReLU zeroes the negative half. The values it passes up therefore have mean square $m_\ell = \tfrac{1}{2}\,\text{Var}(z)$.

Task: write signal_spread(layer_sizes, scheme, input_spread), returning [weight_spreads, signal_spreads]. Each list has one entry per layer, $\ell = 1$ to $L$:

• weight_spreads: $\sigma_\ell$,
• signal_spreads: the spread $\sqrt{\text{Var}(z)}$ of layer $\ell$'s values before its ReLU.

Round every number to 4 decimal places. layer_sizes holds at least two positive whole numbers, scheme is "he" or "xavier", and input_spread is positive.

Worked through, signal_spread([6, 3, 3], "xavier", 1.0):

• Layer 1: $\sigma_1 = \sqrt{2/9} = 0.4714$. $\text{Var}(z) = 6 \times \tfrac{2}{9} \times 1 = 1.3333$, so the spread is $1.1547$. The ReLU passes up $m_1 = 0.6667$.
• Layer 2: $\sigma_2 = \sqrt{2/6} = 0.5774$. $\text{Var}(z) = 3 \times \tfrac{1}{3} \times 0.6667 = 0.6667$, so the spread is $0.8165$.
• The answer is [[0.4714, 0.5774], [1.1547, 0.8165]].

Every layer multiplies the signal's variance by some factor, and a deep network applies its factor again at every layer. This is the compounding from Chapter 3, and it happens before a single gradient has been computed. A factor of $0.9$ looks harmless for one layer. Over fifty layers it leaves $0.9^{50} \approx 0.005$ of the variance. Run a deep stack under both schemes and compare the second lists.