One Map Each

Global average pooling crushes each map down to a single number, so by the time the dense layer sees them, nothing survives in a map that could identify anything at all.

There are $512$ maps and $1000$ classes, so the pairing cannot exist; a deep filter answers to a part that recurs across unrelated categories, and a class score is a learned weighting over all $512$.

Filters at every depth respond only to edges, orientations and colour patches, so no filter anywhere in the network is tied to anything as specific as a class.

The dense layer holds $512 \times 1000 = 512{,}000$ weights, enough on its own to store what separates $1000$ categories, so the class knowledge must live there rather than in the filters.