A short list of numbers, the hidden state, is a recurrent network's whole memory of what it has read so far. It is a summary, not a copy of the words.
The hidden state is a short list of numbers — the network's entire memory of everything it has read so far.
Note what it is not: a copy of the words. It is a compressed summary, and the network decides what to keep.
