Down to Thirty-Two and Back

The bottleneck is a $32 \times 32$ grid, so a boundary read from it is placed only to within $32$ pixels, and no skip connection can improve on that, because detail destroyed by downsampling cannot be recovered anywhere in the network.

The decoder upsamples all the way back to $512 \times 512$, so full resolution — and with it the original detail — is restored by the decoder alone; the skip connections are there to make training converge faster.

One bottleneck cell stands for a $16 \times 16$ block of input, so that route alone fixes a boundary only to roughly $16$ pixels; the $256 \times 256$ skip carries detail at a $2$-pixel scale, and that is what brings $4$ pixels within reach.

Both routes are already finer than $4$ pixels — the bottleneck at $32 \times 32$ and the finest skip at $256 \times 256$ — so the requirement is comfortably met whether or not the skip connections are present at all.