ReLU makes a hard decision at zero: keep the input, or discard it entirely. GELU makes a soft one, weighting each input by roughly how likely it is to be positive — which turns out to work better in transformers, where it's now the default.
The standard tanh approximation:
GELU(x) = 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x³)))
Task: write gelu(values) applying this element-wise, rounded to 4 decimal places.
Look at what happens around −1: the output is slightly negative, then rises back toward zero. GELU isn't monotonic, and that small dip is the part ReLU can't reproduce — a mildly negative input isn't simply deleted, it's given a small negative weight.