The safe first pick for updating weights is Adam, with a learning rate of 0.001. It works well enough on almost any problem without tuning.
This is the honest default. It works acceptably on almost any problem without tuning, which makes it the right thing to try first — and the right thing to move away from only when you have a reason.
