The Company It Keeps

Word2vec rests on one idea: a word is characterised by the words around it. To use that idea you first need the neighbourhood itself as a vector — the average of the embeddings of the tokens surrounding a position, with that position's own token left out.

Task: write context_vectors(embeddings, tokens, window), returning one context vector per position in tokens, each component rounded to 4 decimal places.

• embeddings is the lookup table: row $i$ is the vector for token id $i$.
• tokens is a sequence of token ids. The same id may appear at several positions.
• The context of position $i$ is every position from $i - w$ to $i + w$ except $i$ itself, where $w$ is window.
• Positions that fall outside the sequence simply do not exist. The window is cut short at the ends, never padded.
• The context vector is the elementwise mean of the embedding rows at those context positions.
• If a position has no context at all, return a vector of zeros as wide as a row of the table.

Notice what this vector is not. It is a bag: shuffle the neighbours and it does not change. That is enough to learn which words keep company with which, and it is exactly why it is not enough to read a sentence.