Causal Masking

A language model predicting token 5 must not be allowed to look at token 6. Causal masking enforces that inside attention: before the softmax, every score pointing at a future position is set to -inf, so it comes out of the softmax as exactly zero weight.

Task: write causal_attention(scores), taking a square matrix of raw attention scores and returning the attention weights after masking and softmax, rounded to 4 decimal places.

• Row i may attend to columns 0 through i only.
• Each row's visible entries go through softmax and therefore sum to 1.
• Masked positions are reported as 0.0.
• Row 0 sees only itself, so it's always 1.0 followed by zeros.

Doing it this way is what lets a decoder train on a whole sequence in one pass: every position predicts its own next token simultaneously, and the mask is the only thing preventing each one from reading the answer.