What Survives the Trip Back

Picture the thinnest deep network there is: one neuron per layer, stacked $L$ layers deep. Starting from $h_0 = x$ , each layer computes

z_l = w_l\,h_{l-1} + b_l, \qquad h_l = \text{act}(z_l), \qquad l = 1, \dots, L

and the network's output is $h_L$ .

The backward pass starts at the output with $\dfrac{\partial h_L}{\partial h_L} = 1$ . Every layer it crosses multiplies the signal by that layer's rate, which has two parts, the activation's slope and the weight:

\frac{\partial h_L}{\partial h_{l-1}} = \frac{\partial h_L}{\partial h_l} \cdot \text{act}'(z_l) \cdot w_l

The three activations and their slopes:

`activation`	$\text{act}(z)$	$\text{act}'(z)$
`"sigmoid"`	$\sigma(z) = \dfrac{1}{1 + e^{-z}}$	$\sigma(z)\,\big(1 - \sigma(z)\big)$
`"tanh"`	$\tanh(z)$	$1 - \tanh(z)^2$
`"relu"`	$\max(0, z)$	$1$ if $z > 0$ , otherwise $0$ (including at exactly $z = 0$ )

Task: write stack_gradients(activation, x, weights, biases). Layer $l$ uses weights[l-1] and biases[l-1], and every layer uses the same activation. Run the forward pass, then the backward pass, and return the list

\left[\ \frac{\partial h_L}{\partial h_L},\ \ \frac{\partial h_L}{\partial h_{L-1}},\ \ \dots,\ \ \frac{\partial h_L}{\partial h_0}\ \right]

That is $L + 1$ numbers: 1.0 at the output first, and the gradient that reaches the input $x$ last. Round each one to 4 decimal places. Read left to right, the list shows how much of the signal is still alive after each layer it crosses on the way back.

What Survives the Trip Back

Examples

Hints