A bigram language model predicts the next word from the current one, by counting how often each pair appeared in the training text:
That has a fatal flaw. Any pair that never appeared gets probability exactly zero, so a single unseen pair makes an entire sentence impossible — and in logs, negative infinity.
Add-1 (Laplace) smoothing fixes it by pretending every possible pair was seen once more than it was:
where V is the vocabulary size. The +V on the bottom is there because you added 1 to each of V numerators — without it the row no longer sums to 1.
Task: write bigram_probs(tokens, vocabulary) returning a V × V table where row i, column j is the probability that vocabulary[j] follows vocabulary[i], each rounded to 4 decimal places.
tokens is the training text as a list of words. Count every consecutive pair in it: the pairs are (tokens[0], tokens[1]), (tokens[1], tokens[2]), and so on.count(w1) is how many times w1 appeared as the first word of a pair — which is its total count excluding a final appearance at the very end of the text, since nothing follows it there.vocabulary. Every word in tokens is in the vocabulary.1/V, a uniform guess, which is exactly the right thing to say about a word you've never seen followed by anything.Each row should sum to about 1 (up to rounding) — the quickest way to catch a wrong denominator.