Watching the Words Arrive

A writing assistant shows its reply while it is still being generated. The model predicts one token, appends it, feeds everything back in and predicts the next, and the app puts each token on screen the moment it arrives.

So a user never feels a single latency number. They feel a wait before anything appears, and then a flow of words that can run smoothly or stall.

The app logs, on one clock in milliseconds, the moment the request was sent and the moment each token of the reply arrived.

Task: write stream_metrics(sent_ms, token_ms) and return a tuple of four numbers, each a float rounded to 4 decimal places:

1. First wait: milliseconds from sending the request to the first token arriving. (Serving teams call this time to first token.)
2. Mean gap: the average number of milliseconds between one token and the next, taken over every consecutive pair of tokens in the reply.
3. Worst gap: the largest of those token-to-token gaps.
4. Tokens per second: the number of tokens in the reply divided by the whole wait in seconds, from sending the request to the last token arriving.

• token_ms is in arrival order and always holds at least one token.
• Two tokens can arrive in the same millisecond.
• A one-token reply has no token-to-token gaps, so report its mean gap and worst gap as 0.0.

Each number answers a different complaint. The first wait decides whether the reply feels instant at all. The mean gap is how fast the words flow once they start. The worst gap is the stall in mid-sentence that the average hides.