BM25 Ranking

BM25 is what production search ran on for two decades, and it's still the baseline every neural retriever gets measured against. It fixes two things TF-IDF gets wrong: term frequency should saturate, and long documents shouldn't win just by being long.

For a query term t and document d:

1idf(t)   = ln((N - df + 0.5) / (df + 0.5) + 1)
2
3                    f(t,d) * (k1 + 1)
4score += idf(t) * ---------------------------------------
5                   f(t,d) + k1 * (1 - b + b * |d| / avgdl)

where f(t,d) is the count of t in d, |d| its length, and avgdl the mean document length.

Task: write bm25_scores(documents, query, k1, b) returning one score per document in the original order, rounded to 4 decimal places.

• Documents and the query are lists of tokens.
• A query term appearing in no document contributes nothing — skip it.
• A term repeated in the query is scored each time it appears.

The k1 term is what makes frequency saturate: the tenth occurrence of a word adds far less than the second, because the count sits in both numerator and denominator. With b = 0 length normalization switches off entirely — the last test case is exactly that, and shows the four-occurrence document only modestly ahead of the one-occurrence document.