Matrix-based optimizers have attracted growing interest for improving LLM training efficiency, with significant progress centered on orthogonalization/whitening based methods. While yielding substantial performance gains, a fundamental question arises: can we develop new paradigms beyond orthogonalization, pushing the efficiency frontier further? We present \textbf{Adaptively Rotated Optimization (ARO}, a new matrix optimization framework that treats gradient rotation as a first class design principle. ARO accelerates LLM training by performing normed steepest descent in a rotated coordinate system, where the rotation is determined by a novel norm-informed policy. This perspective yields update rules that go beyond existing orthogonalization and whitening optimizers, improving sample efficiency in practice. To make comparisons reliable, we propose a rigorously controlled benchmarking protocol that reduces confounding and bias. Under this protocol, ARO consistently outperforms AdamW (by 1.3 $\sim$1.35$\times$) and orthogonalization methods (by 1.1$\sim$1.15$\times$) in LLM pretraining at up to 8B activated parameters, and up to $8\times$ overtrain budget, without evidence of diminishing returns. Finally, we discuss how ARO can be reformulated as a symmetry-aware optimizer grounded in rotational symmetries of residual streams, motivating advanced designs that enable computationally efficient exploitation of cross-layer/cross module couplings.

翻译：基于矩阵的优化器在提升大语言模型训练效率方面日益受到关注，其中以正交化/白化方法为核心的进展尤为显著。尽管这些方法带来了显著的性能提升，一个根本性问题随之产生：我们能否超越正交化范式，进一步拓展效率边界？本文提出**自适应旋转优化（ARO）**——一种将梯度旋转作为核心设计原则的新型矩阵优化框架。ARO通过在旋转坐标系中执行归一化最速下降来加速大语言模型训练，其旋转策略由创新的范数信息驱动。这一视角催生了超越现有正交化与白化优化器的更新规则，在实践中提升了样本效率。为确保比较的可靠性，我们提出严格受控的基准测试协议以降低混杂因素与偏差。在该协议下，ARO在激活参数高达80亿、过拟合预算达8倍的LLM预训练中，始终优于AdamW（1.3∼1.35倍）和正交化方法（1.1∼1.15倍），且未观察到收益递减现象。最后，我们探讨了如何将ARO重构为基于残差流旋转对称性的对称感知优化器，这为开发能高效利用跨层/跨模块耦合关系的计算优化设计提供了理论动机。