We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

翻译：我们提出Pion，一种基于正交等价变换的大语言模型（LLM）训练谱保持优化器。与Adam、Muon等加法型优化器不同，Pion通过左右正交变换更新每个权重矩阵，在整个训练过程中保持其奇异值不变。这一优化机制在保持权重矩阵谱范数固定的同时，可对其几何结构进行调控。我们推导了Pion的更新规则，系统分析了其设计选择，并结合多项关键性质研究了其收敛行为。实验结果表明，在LLM预训练与微调中，Pion为标准优化器提供了一种稳定且具有竞争力的替代方案。