Every Dial Set to 0.1

A classifier takes 4 input features, passes them through two hidden layers of 16 neurons each, both using ReLU, and ends in an output layer of 3 neurons, one per class, feeding a softmax. Every layer is fully connected, so the network has

$(4 \times 16 + 16) + (16 \times 16 + 16) + (16 \times 3 + 3) = 403$

weights and biases.

Instead of random initialization, someone sets every weight to $0.1$ and every bias to $0$. The network is then trained with mini-batch gradient descent for many epochs on a varied training set: the examples differ from one another, and all three classes appear.

Backpropagation (Chapter 3) supplies the gradients. In the backward pass each neuron receives a share of the blame, and these three facts describe how it is used:

1. A weight's gradient is the blame at the neuron the weight feeds into, multiplied by the value that travelled along that weight in the forward pass. A bias's gradient is the blame at its own neuron.
2. The output neurons' blame comes straight from the loss. It depends on the network's outputs and on the example's true label, so the true class's output neuron receives different blame from the other two.
3. A hidden neuron's blame is its ReLU slope multiplied by the sum, over every neuron it feeds, of (connecting weight $\times$ that neuron's blame).

With a mini-batch, each gradient is the average of these per-example gradients over the batch.

However long it trains, some of the 403 parameters are forced to stay exactly equal to one another. What is the largest number of different values that can ever appear among the network's 403 weights and biases?

The answer is a whole number.