What the Savings Buy

A text classifier uses an LSTM over 120-dimensional embeddings, with a hidden state of 180 units.

Inside both an LSTM and a GRU, every gate and every candidate is computed the same way — from the current token and the previous hidden state:

$\phi\!\left(W_x\, x_t + W_h\, h_{t-1} + b\right)$

each with its own $W_x$, its own $W_h$ and its own bias $b$. ($\phi$ is the squashing function and holds no parameters.) Call one of these a block.

• An LSTM has four blocks: forget, input, output, and the candidate.
• A GRU has three: update, reset, and the candidate.

Nothing else in either cell holds parameters. So for input size $x$ and hidden size $h$, one block holds $h \times x$ numbers in $W_x$, $h \times h$ numbers in $W_h$, and $h$ biases.

You now replace the LSTM with a GRU and spend the entire saving on a wider hidden state: the GRU must use exactly the same total number of parameters as the LSTM it replaces.

What hidden size does the GRU get?

The answer is a whole number. (The embedding table and the output layer sit outside this count and are unchanged.)