A Landscape That Mostly Curves Up

Training has stalled at a flat point: the gradient is zero in every direction. Whether that point is a dead end depends on how the loss curves as you move away from it.

Here is a crude but classic toy model of a network with $n$ weights. At a flat point, look along $n$ separate directions, one for each weight. Along each direction the loss either curves up (stepping away along it, either way, raises the loss) or curves down (stepping away along it lowers the loss). In the model:

• each direction curves up with probability $0.99$ and down with probability $0.01$;
• the $n$ directions behave independently of one another.

The flat point is then one of the lesson's three shapes:

What is the smallest number of weights $n$ for which a flat point in this model is more likely to be a saddle point than a local minimum?

Give a whole number.