Add, Then Level

Every sublayer in a transformer block hands its result to the same two-step wrapper: add, then normalize. The sublayer's output is added back onto the vectors that went into it — nothing is replaced — and the sum is rescaled so the numbers cannot drift as the stack gets deeper.

Task: write add_and_norm(x, sublayer_out, gamma, beta), returning the wrapped output, rounded to 4 decimal places.

Take one position $i$ at a time. Write $u_i$ for sublayer_out[i], and work from that position's own $d$ slots:

$r = x_i + u_i, \qquad m = \frac{1}{d}\sum_{j=1}^{d} r_j, \qquad s = \sqrt{\frac{1}{d}\sum_{j=1}^{d} (r_j - m)^2 + 10^{-5}}$

$\text{output}_{ij} = \gamma_j \cdot \frac{r_j - m}{s} + \beta_j$

• The spread divides the squared deviations by $d$, not by $d - 1$.
• The $10^{-5}$ inside the square root is there so a position whose slots happen to be all equal does not divide by zero.
• $\gamma$ and $\beta$ are the same two vectors for every position.

This is the "add & norm" that appears twice in every transformer block. The add is what gives the original vectors a clear path through all ninety-six layers instead of being shrunk away; the norm is what stops a harmless-looking drift of a third per layer compounding into numbers no optimizer can work with.