Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL

翻译：动量驱动的黎曼子流形优化在计算上具有挑战性，因为为确保迭代点始终位于子流形上，通常需要求解复杂的微分方程。本文针对一类具有仿射不变度量的稀疏或结构化对称正定矩阵，简化了此类难题。我们提出了一种广义黎曼正规坐标，该坐标能动态正交化度量，并将问题局部转化为欧几里得空间中的无约束优化问题。利用该方法，我们简化了结构化协方差的现有求解方案，并开发了仅需矩阵乘法的无矩阵逆二阶优化器，适用于低精度深度学习。代码：https://github.com/yorkerlin/StructuredNGD-DL