Back Along the Chain

A plain RNN with a one-number hidden state reads $T$ tokens and makes a single prediction at the end.

The forward pass, for $t = 1, 2, \dots, T$:

$a_t = w\, h_{t-1} + u\, x_t + b, \qquad h_t = \tanh(a_t)$

$h_0$ is a fixed starting value, handed to the network rather than learned. The prediction is the final hidden state $h_T$, and the loss is its squared error against a target $y$:

$L = (h_T - y)^2$

Only three numbers are learned — $w$, $u$ and $b$ — and the same three are used at every one of the $T$ steps.

Task: write bptt_grads(tokens, w, u, b, h0, target) returning [dL_dw, dL_du], the gradient of the loss with respect to $w$ and with respect to $u$, in that order, rounded to 4 decimal places.

You will need the derivative of the activation,

$\frac{d}{da}\tanh(a) = 1 - \tanh^2(a),$

which, because $h_t = \tanh(a_t)$, is simply $1 - h_t^2$ at step $t$.

• The loss exists only at the last step. There is no loss term at any earlier step.
• tokens is a list of $T$ numbers, with $T \ge 1$.
• Nothing before the first token is learnable, so the backward pass has somewhere to stop.
• Round only the two numbers you return; keep full precision everywhere in between.

Nothing in this is a new algorithm. It is the chain rule, run along the unrolled chain instead of down a stack of layers. What it buys you is a direct look at the chapter's central claim: feed it a long sequence with $|w|$ well below $1$ and watch how little of the loss's influence is left by the time it reaches the first token.