A tagger sorts every token of a corpus into one of three buckets: noun, verb, other. Counting over the whole corpus gives the true distribution
A crude baseline model, which never looks at context, always predicts
You want to know how many extra bits per token this baseline wastes compared with a coder that knew exactly — that is, the KL divergence measured in bits.
Base-2 logarithms (you will not need all of them):
What is , in bits? Round your answer to 3 decimal places.