One Part of the Recipe Swapped

Adam's two memories start empty, which makes its first few steps come out far too large. That start-up effect, not the learning rate, is what blew up the run.

Adam moves each weight by up to about the learning rate itself per update, whatever the gradient's size, so $0.1$ is roughly 200 times the recipe's movement. Restart Adam near $0.001$.

A blow-up inside the first 20 updates is the signature of a missing warmup. Keep $0.1$, but ramp up to it from a very low rate over the first few hundred updates.

Adam divides each gradient by its recent size, and these gradients are small, so its steps come out smaller than SGD's. The learning rate cannot be the cause, so look for a bug in the data pipeline instead.