Only the Scale

Layer normalization does two things to each position's vector. It recentres it (subtract the mean) and rescales it (divide by the spread), then applies a learned scale and shift. Many recent language models keep only the rescaling. Their version, RMSNorm, divides each vector by its root mean square and multiplies by a learned gain, one per slot. No mean is subtracted, and there is no shift.

For a vector $\mathbf{x}$ of length $d$ and gains $g_1, \dots, g_d$ :

\text{rms}(\mathbf{x}) = \sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^{2} + \epsilon}, \qquad y_i = g_i \cdot \frac{x_i}{\text{rms}(\mathbf{x})}

Here $\epsilon$ is a tiny number that prevents a division by zero when every slot is $0$ . It sits inside the square root.

Like layer norm, RMSNorm works on each position's vector separately: one root mean square per vector, computed from that vector alone.

Task: write rms_norm(vectors, gain, eps), where vectors is a list of position vectors, all of length $d$ , and gain is a list of $d$ gains. Return the normalized vectors, with every number rounded to 4 decimal places.

Dropping the recentring saves work on every vector in every layer, and in practice it works about as well. The rescaling is the part that keeps the numbers from drifting through a deep stack.

Only the Scale

Examples

Hints