Incognition logo
Home
Learn
Practice
Leaderboard
0
VisualizationsSequences, Language & Attention

Why RNNs Forget

To choose "was" over "were", a model must still remember "cat" eight steps later. Slide to change the distance between the two highlighted words.

The
cat
,
which
we
adopted
last
spring
,
was
"cat" decides "was" — eight tokens later
distance

To choose “was” rather than “were”, the model must still be holding “cat” eight steps later.

Customize