Attention can only mix words by averaging them. Click What is missing? to see what the feed-forward layer adds: reshaping each vector in a way no average can.
Averaging, however cleverly weighted, is still averaging. Stacking pure attention would never give you a non-linear model.
