Incognition logo
Home
Learn
Practice
Leaderboard
0
VisualizationsSequences, Language & Attention

Multi-Head Attention

One attention pattern for "slept" puts most of its weight on "cat". Its weights add up to 1, so it can't also focus hard on "quietly".

one attention pattern for “slept”
the
tired
cat
slept
quietly

A single attention pattern must choose. Here it found the subject — but “slept” also relates to “quietly”, and one set of weights summing to 1 cannot express both strongly.

Customize