Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL

翻译：带动量的黎曼子流形优化在计算上具有挑战性，因为为确保迭代点保持在子流形上，通常需要求解复杂的微分方程。本文针对一类具有仿射不变度量的稀疏或结构化对称正定矩阵，简化了此类困难。我们提出了一种广义版本的黎曼正规坐标，该坐标动态正交化度量，并将问题局部转化为欧几里得空间中的无约束问题。利用该方法，我们简化了结构化协方差的现有方法，并仅通过矩阵乘法开发了一种无需矩阵逆运算的低精度深度学习二阶优化器。代码：https://github.com/yorkerlin/StructuredNGD-DL