Two Ways to Tame a Spike

A small model has trained calmly for hundreds of steps. Then one batch produces a gradient spike, and the backward pass returns this gradient for the model's four weights:

$g = (0.6,\ -9.0,\ 0.8,\ 12.0)$

Two teammates each propose a clipping rule. Both use the same limit, $1$ .

rule	what it does to $g$
A: clip by norm	if $\lVert g \rVert > 1$ , rescale the whole vector: $g \leftarrow g \cdot \dfrac{1}{\lVert g \rVert}$
B: clip by value	clamp each entry separately into $[-1,\ 1]$ : entries above $1$ become $1$ , entries below $-1$ become $-1$ , the rest are left alone

Here $\lVert g \rVert = \sqrt{g_1^2 + g_2^2 + g_3^2 + g_4^2}$ is the ordinary length of the vector. Whichever rule is used, its output replaces $g$ in the update, and the weights move against it.

What is the angle, in degrees, between the vector that Rule A produces and the vector that Rule B produces?

Round to 1 decimal place. Carry at least four decimals through the intermediate steps.

Two Ways to Tame a Spike

Hints