How Long to Hold the Door

A photo-tagging service runs one copy of its model on one GPU. Requests arrive steadily at 400 per second, evenly spaced (one every $2.5$ ms), all day long.

Timing the forward pass at different batch sizes gave a simple rule:

$\text{time to run a batch of } b \text{ requests} = 40\text{ ms} + 1\text{ ms} \times b$

The $40$ ms is paid whatever the batch size; each extra request in the batch adds only $1$ ms.

The server batches using a fixed collection window of $W$ milliseconds:

1. It collects every request that arrives during the window.
2. When the window closes, that group becomes one batch and joins the GPU's queue, and the next window opens immediately.
3. The GPU runs queued batches one at a time, oldest first, each exactly as collected.

Setting $W = 0$ means no batching at all: every request is run on its own, as a batch of one.

The product team has two requirements:

• the server must keep up with the traffic indefinitely, and
• no request may take more than 300 ms from arriving to being answered.

Treat network, preprocessing and postprocessing time as zero. Which collection window meets both requirements?