Epoch After Epoch

Chapter 5 ended with a full training run: forward pass, loss, backward pass, update, batch after batch and epoch after epoch. Here you write that loop yourself, for the smallest network that still has a hidden layer, with nothing random in it so every run is reproducible.

The network. An input $x$ has $d$ numbers. A hidden layer of $h$ ReLU neurons feeds a single output with no activation:

z_j = \sum_{i} x_i \, W^{(1)}_{ij} + b^{(1)}_j, \qquad a_j = \max(0, z_j), \qquad \hat{y} = \sum_{j} a_j \, W^{(2)}_j + b^{(2)}

In code these are W1[i][j] (the weight from input $i$ to hidden neuron $j$ ), b1[j], W2[j] (one number per hidden neuron) and b2 (a single number).

The ReLU's slope. When you send gradients back through a hidden neuron, its slope is $1$ where $z_j > 0$ and $0$ where $z_j \le 0$ . That includes $z_j$ being exactly $0$ : there the slope is $0$ .

The loss. Mean squared error: the average of $(\hat{y} - y)^2$ over whichever examples are being scored. For a batch of $m$ rows, each row's prediction receives $\partial L / \partial \hat{y} = \tfrac{2}{m}(\hat{y} - y)$ ; every other gradient follows from the chain rule.

The run.

Mini-batches are taken in order, with no shuffling: rows 0 to batch_size - 1, then the next batch_size rows, and so on. If the rows don't divide evenly, the last batch is simply smaller.
For each batch, compute the gradient of that batch's mean squared error with respect to every weight and bias, then update all of them with plain gradient descent: $\theta \leftarrow \theta - \text{lr} \cdot \dfrac{\partial L}{\partial \theta}$ .
One epoch is one pass through every batch.

Task: write train(X, y, W1, b1, W2, b2, lr, batch_size, epochs). X is a list of rows and y a list of targets.

Return a list with one number per epoch: the mean squared error over the whole training set, measured with the weights as they stand at the end of that epoch, rounded to 4 decimal places.

Epoch After Epoch

Examples

Hints