The green square is one output of a 3×3 layer. The purple patch around it marks every input pixel that can change it: its receptive field.
The green square is one unit in the output. The purple region is every input pixel that can influence it — its receptive field. After one 3×3 layer, it is 3×3.
