BERT lets "bank" in "the bank by the river" see words on both sides. Click to look left only, and there is no way to tell which bank is meant.
BERT drops the mask entirely. Every token sees the whole sentence, which is exactly what you want when you are reading rather than writing.
