Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
arxiv(2023)
摘要
Riemannian submanifold optimization with momentum is computationally
challenging because, to ensure that the iterates remain on the submanifold, we
often need to solve difficult differential equations. Here, we simplify such
difficulties for a class of sparse or structured symmetric positive-definite
matrices with the affine-invariant metric. We do so by proposing a generalized
version of the Riemannian normal coordinates that dynamically orthonormalizes
the metric and locally converts the problem into an unconstrained problem in
the Euclidean space. We use our approach to simplify existing approaches for
structured covariances and develop matrix-inverse-free 2^nd-order
optimizers for deep learning with low precision by using only matrix
multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要