Every Input to Every Neuron

Put several neurons side by side, connect every input to every one of them, and you have a layer. Because every input is wired to every neuron, this kind is called a dense, or fully connected, layer. A layer with $3$ inputs and $2$ neurons has $3 \times 2 = 6$ connections, so it has $6$ weights, plus one bias for each neuron.

Those weights have to be stored somehow. Here they come as a connection table with one row per input and one column per neuron: weights[i][j] is the weight on the connection from input i to neuron j. For that 3-input, 2-neuron layer:

Neuron j takes each input $x_i$ of an example, multiplies it by the weight on the connection from input i to neuron j, adds up those products, and then adds its own bias biases[j]. That whole total, bias included, is the neuron's $z$:

$z_j = \sum_i x_i \cdot \text{weights}[i][j] + \text{biases}[j]$

The neuron then outputs $\text{ReLU}(z_j) = \max(0, z_j)$, so the bias goes in before the clipping.

Task: write dense_forward(batch, weights, biases), which runs a whole batch through one such layer.

• batch is a list of n examples, and each example is a list of d input values.
• weights is the d × k connection table for a layer of k neurons.
• biases has k entries, one per neuron.
• Return an n × k list of lists, with one row per example and one entry per neuron. Round each value to 4 decimal places. A layer with a single neuron still returns one row per example, and each row is still a list.

Each row you return is a new description of that example, written in the layer's own k features instead of the d it arrived with, and it is exactly what the next layer would take as its input. Nobody designs those features by hand: they are whatever the learned weights make them.