Queries From One Side

The encoder has read the French sentence and produced one vector per input word. The decoder is part-way through writing the English, with its own vector for each output position so far. Cross-attention is the only wire between the two stacks, and it is ordinary attention with one change: the query comes from the decoder, the keys and the values from the encoder.

Task: write cross_attention(decoder_states, encoder_outputs, Wq, Wk, Wv), returning one context vector per decoder position, rounded to 4 decimal places.

Project, score, scale, softmax, blend:

$Q = D W_q, \qquad K = E W_k, \qquad V = E W_v$

$\text{score}_{ts} = \frac{Q_t \cdot K_s}{\sqrt{d_k}}$

Softmax each decoder row across all $S$ encoder positions, giving weights $w_{ts}$ that sum to $1$ for each $t$, then

$\text{context}_t = \sum_{s=1}^{S} w_{ts}\, V_s$

Return the $T$ context vectors, each of length $d_v$. There is no mask anywhere in this operation: the encoder has already read the entire input, so every output position is free to draw on all of it.

Nothing here is a new mechanism. Point all three projections at a single sequence instead of two and this identical code is self-attention — which is why one block can serve translation, classification and generation alike.