Two Matrices, Swapped

A self-attention layer projects each token $x_i$ into three vectors using three learned matrices

$q_i = W_Q x_i, \qquad k_i = W_K x_i, \qquad v_i = W_V x_i$

and then computes, for the query at position $i$,

$\alpha_{ij} = \operatorname{softmax}_j\!\left(\frac{q_i \cdot k_j}{\sqrt{d}}\right), \qquad \text{out}_i = \sum_j \alpha_{ij}\, v_j$

An engineer refactoring the code swaps two lines. The tensor produced by $W_K$ is now handed to the layer as the values, and the tensor produced by $W_V$ is handed to it as the keys. The swap is consistent everywhere, both matrices produce vectors of the same width $d$, and the layer is then trained from a fresh random initialisation on the usual task. So the refactored layer computes

$\alpha_{ij} = \operatorname{softmax}_j\!\left(\frac{q_i \cdot v_j}{\sqrt{d}}\right), \qquad \text{out}_i = \sum_j \alpha_{ij}\, k_j$

What happens when this layer is trained?