The Whole Trip Back

This is the backward pass on a real, if tiny, network, from start to finish: run it forward, score it, then find a gradient for every weight and every bias in it.

The network. $d$ inputs feed a hidden layer of $h$ neurons with ReLU, which feeds an output layer of $k$ neurons with no activation. Both layers are dense and use a connection table: W[i][j] is the weight on the wire from unit i of the layer below to neuron j.

On one example $x$, the forward pass computes

$z_j = \sum_i x_i \, \text{W1}[i][j] + \text{b1}[j], \qquad a_j = \max(0, z_j), \qquad \hat y_m = \sum_j a_j \, \text{W2}[j][m] + \text{b2}[m]$

The loss is the mean squared error over the whole batch. Square the error of every output on every example, add them up, and divide by how many there were:

$L = \frac{1}{n\,k} \sum_{e=1}^{n} \sum_{m=1}^{k} \left(\hat y_{e,m} - y_{e,m}\right)^2$

ReLU's slope is $1$ where $z > 0$ and $0$ where $z \le 0$, including at exactly $z = 0$.

Task: write backward_pass(inputs, targets, W1, b1, W2, b2), returning

[loss, dW1, db1, dW2, db2]

where loss is $L$ and each gradient holds $\dfrac{\partial L}{\partial(\cdot)}$ for every entry of the parameter it is named after, laid out exactly like that parameter (dW1 is $d \times h$, db2 has $k$ entries, and so on).

• inputs has $n \ge 1$ examples of $d$ values; targets has the same $n$ rows of $k$ values.
• Round every number to 4 decimal places, and only at the end. A table with a single row or a single column is still a list of lists.

For the first test, the forward pass gives $z = [1.0,\ 0.5]$, so $a = [1.0,\ 0.5]$ and $\hat y = 2.0$ against a target of $1.0$. The loss is therefore $1.0$. The gradients are yours to find.

Once these gradients exist, the update is one line per parameter: step each one against its own gradient. They all move together, because every gradient here came from the same forward pass, with the weights as they stood then. A classic bug in hand-written training loops updates W2 first and then sends the error back through the new W2. That hands the hidden layer blame for a network that never made the prediction.