The Bill for One Step

A network that reads handwritten digits takes 500 inputs (pixel values), has two hidden layers of 200 and 100 ReLU neurons, and ends in 10 output neurons with no activation. Every layer is fully connected, and to keep the count clean the network has no biases.

Making one prediction takes exactly one multiplication per weight, since every weight multiplies the one value that arrives along it. That is $121{,}000$ multiplications.

Now count one full training step on one example: a forward pass, the loss, a backward pass that finds the gradient of every weight, and then the update, which multiplies each weight's gradient by the learning rate $\eta$ and subtracts it from the weight. Count every multiplication of two numbers, with these rules:

• Computing the loss and its slope at the outputs costs nothing.
• Applying ReLU costs nothing, and so does multiplying by ReLU's slope of $0$ or $1$ on the way back.
• Every multiplication is carried out in full, even when one of its numbers happens to be $0$.
• Nothing is worked out that no weight's gradient could ever depend on.

How many multiplications does the whole training step take?