A self-attention head of width is reading the sentence "the cat sat down". The token "sat" supplies the query, and its query has already been dotted with all four keys. These are the raw, unscaled dot products:
| token | |
|---|---|
| the | |
| cat | |
| sat | |
| down |
These are not weights yet. Finish the two remaining operations of the formula that stand between a row of raw dot products and a row of attention weights.
What attention weight does "sat" end up placing on itself?
Round your answer to 3 decimal places.