Back Through Batch Norm

Batch normalization standardises every feature across the batch, then lets the network rescale and shift it. Running it forwards is a few lines. This problem is about running it backwards.

A batch holds $N$ examples with $D$ features each, and x[i][j] is feature $j$ of example $i$ . In training mode, the forward pass works column by column, using statistics computed from this batch:

\mu_j = \frac{1}{N}\sum_{i} x_{ij}, \qquad \sigma_j^2 = \frac{1}{N}\sum_{i} \left(x_{ij} - \mu_j\right)^2

\hat{x}_{ij} = \frac{x_{ij} - \mu_j}{\sqrt{\sigma_j^2 + \varepsilon}}, \qquad y_{ij} = \gamma_j\,\hat{x}_{ij} + \beta_j

The variance divides by $N$ , not $N - 1$ . The rest of the network turns the outputs $y$ into a loss $L$ , and the backward pass has already delivered the upstream gradient dout[i][j] $= \dfrac{\partial L}{\partial y_{ij}}$ .

Task: write batchnorm_backward(x, dout, gamma, eps) and return a tuple (dx, dgamma, dbeta):

dx: an $N \times D$ list of lists with dx[i][j] $= \dfrac{\partial L}{\partial x_{ij}}$ ,
dgamma: a list of $D$ values, $\dfrac{\partial L}{\partial \gamma_j}$ ,
dbeta: a list of $D$ values, $\dfrac{\partial L}{\partial \beta_j}$ .

Round every number to 4 decimal places. eps is always positive, and $N$ can be as small as 1.

Back Through Batch Norm

Examples

Hints