Blame on Every Wire

During the backward pass every layer is handed an error signal: one number per neuron, saying how fast the loss changes as that neuron's weighted sum $z$ changes. Receiving it is only half of the layer's job. The layer still has to turn it into a gradient for each of its own weights and biases, one per connection, so that every one of them can be updated.

The layer is dense and its weights are stored as a connection table, with one row per input and one column per neuron: weights[i][j] is the weight on the wire from input i to neuron j. With $d$ inputs and $k$ neurons the table is $d \times k$. On one example, neuron $j$ computes

$z_j = \sum_i x_i \cdot \text{weights}[i][j] + \text{biases}[j]$

The training step runs a batch of $n$ examples, and the loss it minimises is the average of the examples' own losses:

$L = \frac{1}{n}\sum_{e=1}^{n} L_e$

For each example $e$ you are given two lists:

• inputs[e]: the $d$ values $x_i$ that arrived at the layer on that example. The forward pass stored them.
• errors[e]: the $k$ error signals the backward pass delivered for that example alone. Entry $j$ is $\dfrac{\partial L_e}{\partial z_j}$, how fast example $e$'s own loss changes as neuron $j$'s weighted sum on that example changes.

Task: write layer_gradients(inputs, errors), returning [weight_grads, bias_grads]:

• weight_grads: a $d \times k$ table in the same layout as weights. Entry [i][j] is $\dfrac{\partial L}{\partial\, \text{weights}[i][j]}$, the gradient of the batch loss $L$.
• bias_grads: $k$ entries, where entry $j$ is $\dfrac{\partial L}{\partial\, \text{biases}[j]}$.

Round every value to 4 decimal places. inputs has $n \ge 1$ rows of $d$ numbers, and errors has the same $n$ rows, each of $k$ numbers. A table with a single row or a single column is still a list of lists.

Notice what this function was never given: the layer's weights. Everything it needed was either stored on the way forward or delivered on the way back. That is why a training framework keeps every layer's input in memory until the backward pass has come through. The forward pass cannot throw them away as soon as the next layer has used them.