One Layer's Share of the Blame

A hidden layer takes 4 inputs and has 3 ReLU neurons, so it holds $4 \times 3 = 12$ weights and $3$ biases: 15 parameters in all. Neuron $i$ computes the weighted sum

$z_i = w_{i1}a_1 + w_{i2}a_2 + w_{i3}a_3 + w_{i4}a_4 + b_i$

and sends $\text{ReLU}(z_i)$ on to the next layer. So $w_{ij}$ is the weight that carries input $a_j$ into neuron $i$.

During the forward pass on one training example, the layer cached its inputs and its weighted sums. Here are the inputs:

The two zeros come from neurons in the layer below that fell silent.

The backward pass has already worked its way from the output down to this layer. The last column below is the blame it delivered to each neuron's output, $\dfrac{\partial L}{\partial\, \text{ReLU}(z_i)}$: how much the loss changes per unit nudge of what that neuron sent onward.

The backward pass now computes a gradient for every one of the layer's 15 parameters.

How many of the 15 parameters get a gradient of exactly zero on this example?

The answer is a whole number.