Attention has no idea what order its inputs came in — shuffle the tokens and it computes the same thing. Positional encoding is the fix: add a distinct, position-dependent vector to every token embedding.
For position pos and dimension index i in a model of width d_model:
| index | value |
|---|---|
even i | sin(pos / 10000^(2⌊i/2⌋ / d_model)) |
odd i | cos(pos / 10000^(2⌊i/2⌋ / d_model)) |
Note the ⌊i/2⌋ — dimensions 0 and 1 share a frequency, 2 and 3 share the next, and so on. Each sin/cos pair is one rotating clock hand.
Task: write positional_encoding(seq_len, d_model) returning a seq_len × d_model table, every value rounded to 4 decimal places.
seq_len = 0 returns an empty list.0.The wavelengths run from 2π up to about 10000·2π. Low dimensions flip quickly and pin down local order; high dimensions barely move across the whole sequence and encode roughly where you are overall.