Before the Dot Product

Chapter 3 scores a query against a key with a dot product. The first attention models for translation scored differently. They passed the decoder's current state and each encoder state through a small learned layer of their own, and read a single number out of the result. This is called additive attention.

The decoder state $\mathbf{s}$ is doing the asking. The encoder states $\mathbf{h}_1, \dots, \mathbf{h}_n$ are the input words it can look at.

Score. Two learned matrices project the decoder state and each encoder state into the same number of slots, $a$ . The two projections are added slot by slot, each slot is squashed with $\tanh$ , and a learned vector $\mathbf{v}$ of length $a$ combines the slots into one number:

e_j = \sum_{i=1}^{a} v_i \tanh\big( (W_{\text{dec}}\,\mathbf{s})_i + (W_{\text{enc}}\,\mathbf{h}_j)_i \big)

Because both projections land in $a$ slots, the decoder state and the encoder states do not have to be the same size. A dot product between them would demand that.

Weights. Softmax over the scores $e_1, \dots, e_n$ gives attention weights $\alpha_1, \dots, \alpha_n$ that are positive and sum to 1.

Context. The context vector is the weighted sum of the encoder states themselves:

\mathbf{c} = \sum_{j=1}^{n} \alpha_j \mathbf{h}_j

Matrices are given as lists of rows: W[i] is row $i$ , so $(W\mathbf{x})_i$ is sum(W[i][k] * x[k] for k in range(len(x))).

Task: write additive_attention(decoder_state, encoder_states, W_dec, W_enc, v). Return a tuple (weights, context): the list of attention weights, one per encoder state, and the context vector, the same size as one encoder state. Round every number to 4 decimal places, and build the context from the unrounded weights.

Before the Dot Product

Examples

Hints