Incognition logo
Home
Learn
Practice
Leaderboard
0
VisualizationsSequences, Language & Attention

Masked Attention

A model guessing "sat" can see the whole sentence, answer included, so it learns nothing. Click Block the future to hide the words ahead.

predicting word 3, “sat”
the
cat
sat
down
the answer is right there in the input — nothing is being learned

If a model can see the next word while guessing it, training is meaningless. It scores perfectly and learns nothing.

Customize