Gradient Clipping by Global Norm

One bad batch can produce an enormous gradient, and a single step along it wrecks weights that took hours to train — the loss goes to NaN and the run is over. Clipping caps the step size without changing where it points.

1global_norm = sqrt(sum of every gradient element squared, across all tensors)
2
3if global_norm > max_norm:
4    every element *= max_norm / global_norm

Task: write clip_by_global_norm(gradients, max_norm), where gradients is a list of tensors (each a flat list). Return the same structure, rounded to 4 decimal places.

• The norm is computed once, over everything at once — not per tensor.
• If the norm is already within max_norm, return the gradients unchanged.
• A norm of exactly zero is left alone.

Doing it globally rather than per-tensor is the important part: one shared scale factor preserves the relative sizes between layers, so the update still points in the direction backprop computed. Clipping each tensor separately would quietly rotate it.