Carrying the Cat Through the Clause

One number inside an LSTM's cell state has learned to track a single fact: the subject of this sentence is singular. The model is reading

The cat that we adopted last spring was sleeping on the stairs.

and we follow that one number through four consecutive steps — cat, that, we, adopted.

At each step the cell is edited, and then part of what it holds is revealed as that step's visible output:

$\text{cell} \;\leftarrow\; \text{forget} \times \text{cell} \;+\; \text{input} \times \text{candidate}$

$\text{revealed} \;=\; \text{output} \times \text{cell}$

Just before step 1 the number is $0$ — nothing has been stored yet. The candidate the cell computes and the three gate values it produces at each step are:

step	token	forget	candidate	input	output
1	cat	0.5	1.0	0.8	0.4
2	that	1.0	0.9	0.0	0.1
3	we	1.0	0.4	0.25	0.3
4	adopted	0.9	0.2	0.5	0.5

What value does this number reveal at step 4?

Carrying the Cat Through the Clause

Hints