Word Count Dictionary

Counting words is the first step of nearly every text pipeline. The counting is trivial; the decisions about what counts as "the same word" are the actual work, so here they are, pinned down.

Task: write word_counts(text) returning a dictionary mapping each word to how many times it appears.

Process the text like this:

1. Lowercase the whole thing, so The and the are one word.
2. Split on whitespace. Any amount of spaces, tabs or newlines between words.
3. From each piece, strip these characters off both ends: . , ! ? ; : ' " — so hello, becomes hello.
4. If a piece is left empty after stripping, skip it.
5. Count what remains.

• Strip only from the ends. An apostrophe in the middle stays put, so it's stays it's and is not two words.
• Words appear in the dictionary only if they actually occur; there are no zero counts.

Every one of those rules is a choice, and a different pipeline might choose differently — splitting it's into it and is, or stripping hyphens, or keeping case for proper nouns. That's exactly why real tokenizers are fiddly: the counting never changes, but what you decide to count does.