One Long List of Numbers

When a trained network is saved to disk, it isn't saved layer by layer. Every weight and every bias is written out one after another into one long flat list of numbers. To use the network again you need two things: that list and the layer widths. Nothing in the list marks where one layer's numbers stop and the next layer's start. Only the widths tell you that.

Task: write network_forward(batch, sizes, params). It rebuilds the network from params and runs a batch through it.

• sizes lists the layer widths in order and has at least two entries. sizes[0] is the number of input values. The input layer is just the data arriving, so it has no neurons and no parameters. Every later entry is a layer of neurons, and the last one is the output layer.
• Each neuron connects to every unit in the layer before it, with one weight per connection, and has one bias.
• params holds all of these, layer by layer, starting from the first layer after the input. Within a layer the weights come first, neuron by neuron: all of the first neuron's incoming weights (in the order of the units they come from), then all of the second neuron's, and so on. That layer's biases follow, one per neuron, in neuron order. Then the next layer begins.
• Every hidden layer applies ReLU, $\max(0, z)$, to each neuron's weighted sum. The output layer applies no activation, so its outputs are its $z$ values unchanged.
• batch is a list of examples, and each example has sizes[0] values. Return one row per example, holding the output layer's values rounded to 4 decimal places. Round only at the end.
• params always has exactly the right length for sizes.

For example, with sizes = [3, 2, 1] the list holds $11$ numbers, laid out like this:

The first test is small enough to check by hand: two inputs that are each $0$ or $1$, two hidden neurons and one output. Once your code runs, look at the four answers it gives. No single neuron could produce that column of answers, because no straight line splits those four points into the two groups the network makes. The hidden layer is what bends the boundary.