Holding the Subject

An LSTM is reading

The cat that our neighbours across the street adopted from the shelter last spring …

and has to remember that the subject was singular until its verb finally arrives. When it reads "cat", it writes the value $0.8$ into one slot of its cell state.

From then on, treat the rest of the clause as filler for that slot:

• the input gate stays shut, so nothing new is ever written into it;
• the forget gate's weights contribute nothing, so its value at every step is set by its bias alone: $f = \sigma(b)$, where

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

At every step after "cat", the slot is multiplied by $f$. If the verb arrives $k$ steps after "cat", the slot has been multiplied by $f$ exactly $k$ times.

The team has set the forget-gate bias to $b = 3$. The verb can only use the subject if the slot still holds at least $0.4$ when the verb arrives.

What is the largest number of steps $k$ between "cat" and the verb for which that holds? Give a whole number.