Riemannian submanifold optimization with momentum is computationally challenging because ensuring iterates remain on the submanifold often requires solving or approximating difficult differential equations. We simplify such optimization algorithms for a class of structured symmetric positive-definite matrices with the affine invariant metric. We propose a generalized version of the Riemannian normal coordinates which preserves the metric and dynamically trivializes the problem into a Euclidean unconstrained problem. We use our approach to explain and simplify existing approaches for structured covariances and develop efficient second-order optimizers for training large-scale NNs without matrix inverses.

翻译：具有动量的黎曼子流形优化在计算上具有挑战性，因为确保迭代点始终位于子流形上通常需要求解或近似复杂的微分方程。我们针对一类具有仿射不变度量的结构化对称正定矩阵，简化了此类优化算法。我们提出了一种广义黎曼法坐标，该坐标在保持度量的同时，将问题动态简化为欧几里得无约束问题。利用该方法，我们解释并简化了现有结构化协方差优化方法，并开发了无需矩阵求逆即可高效训练大规模神经网络的二阶优化器。