Spectral optimizers such as Muon have recently shown strong empirical performance in large-scale language model training, but the source and extent of their advantage remain poorly understood. We study this question through the linear associative memory problem, a tractable model for factual recall in transformer-based models. In particular, we go beyond orthogonal embeddings and consider Gaussian inputs and outputs, which allows the number of stored associations to greatly exceed the embedding dimension. Our main result sharply characterizes the recovery rates of one step of Muon and SGD on the logistic regression loss under a power law frequency distribution. We show that the storage capacity of Muon significantly exceeds that of SGD, and moreover Muon saturates at a larger critical batch size. We further analyze the multi-step dynamics under a thresholded gradient approximation and show that Muon achieves a substantially faster initial recovery rate than SGD, while both methods eventually converge to the information-theoretic limit at comparable speeds. Experiments on synthetic tasks validate the predicted scaling laws. Our analysis provides a quantitative understanding of the signal amplification of Muon and lays the groundwork for establishing scaling laws across more practical language modeling tasks and optimizers.

翻译：诸如Muon等频谱优化器近期在大规模语言模型训练中展现了强劲的实证表现，但其优势的来源与程度仍鲜为人知。我们通过线性联想记忆问题（一种适用于Transformer模型事实记忆研究的可解析模型）来探究该问题。具体而言，我们突破正交嵌入的限制，引入高斯输入输出分布，使得存储关联数量可大幅超过嵌入维度。我们的主要结果精确刻画了在幂律频率分布下，单步Muon与SGD在逻辑回归损失上的恢复率特征。研究表明，Muon的存储容量显著超过SGD，且其饱和临界批规模更大。我们进一步分析了阈值梯度近似下的多步动力学过程，发现Muon的初始恢复率显著快于SGD，而两种方法最终均以相近速度收敛至信息论极限。合成任务实验验证了所预测的缩放规律。本分析为理解Muon的信号放大效应提供了定量框架，并为在更实际的建模任务与优化器中建立缩放规律奠定了理论基础。