First-order methods almost always avoid strict saddle points
Mathematical Programming(2019)
摘要
We establish that first-order methods avoid strict saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including (manifold) gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid strict saddle points.
更多查看译文
关键词
Gradient descent, Smooth optimization, Saddle points, Local minimum, Dynamical systems, 90C26
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络