Text Chunking with Overlap

A document that's too long for a model's context window has to be cut into chunks. Cut it into clean, non-overlapping blocks and you'll sometimes slice a sentence — or an idea — straight down the middle, leaving neither chunk able to answer a question about it. So chunks are usually cut with a bit of overlap, repeating the tail of each one at the head of the next.

Task: write chunk_text(tokens, size, overlap) returning the list of chunks.

• Each chunk is size tokens long, taken in order.
• The next chunk starts size - overlap tokens later, so the last overlap tokens of one chunk reappear at the start of the next.
• Keep going until a chunk reaches the end of the tokens. The final chunk may be short — keep it; don't pad it and don't drop it.
• overlap is always smaller than size, so you always make forward progress.
• If size is bigger than the whole list, you get a single chunk holding everything.

With size = 3 and overlap = 1 the chunks start at tokens 0, 2, 4, and so on — each one sharing a single token with its neighbour.

The trap to avoid is the loop condition. Once a chunk has reached the end of the list, stop: advancing again produces chunks that are entirely overlap, repeating tokens you've already emitted. And if overlap ever equalled size, the start position would never move and the loop would never end — which is why that's ruled out above.