Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free $2^\text{nd}$-order optimizers for deep learning in low precision settings. Code: https://github.com/yorkerlin/StructuredNGD-DL

翻译：黎曼子流形优化结合动量在计算上具有挑战性，因为为确保迭代点始终位于子流形上，通常需要求解复杂的微分方程。本文针对一类具有仿射不变度量的结构对称正定矩阵，简化了此类困难。我们通过提出一种广义的黎曼法坐标方法，动态正交化度量并将问题局部转化为欧氏空间中的无约束问题。利用该方法，我们简化了结构化协方差的现有方法，并开发了无需矩阵逆运算的$2^\text{nd}$阶优化器，适用于低精度深度学习场景。代码：https://github.com/yorkerlin/StructuredNGD-DL