Muon updates weight matrices along (approximate) polar factors of the gradients and has shown strong empirical performance in large-scale training. Existing attempts at explaining its performance largely focus on single-step comparisons (on quadratic proxies) and worst-case guarantees that treat the inexactness of the polar-factor as a nuisance ``to be argued away''. We show that already on simple strongly convex functions such as $L(W)=\frac12\|W\|_{\text{F}}^2$, these perspectives are insufficient, suggesting that understanding Muon requires going beyond local proxies and pessimistic worst-case bounds. Instead, our analysis exposes two observations that already affect behavior on simple quadratics and are not well captured by prevailing abstractions: (i) approximation error in the polar step can qualitatively alter discrete-time dynamics and improve reachability and finite-time performance -- an effect practitioners exploit to tune Muon, but that existing theory largely treats as a pure accuracy compromise; and (ii) structural properties of the objective affect finite-budget constants beyond the prevailing conditioning-based explanations. Thus, any general theory covering these cases must either incorporate these ingredients explicitly or explain why they are irrelevant in the regimes of interest.
翻译:Muon沿梯度的(近似)极因子更新权重矩阵,并在大规模训练中展现出强大的实证性能。现有解释其性能的尝试主要集中于单步比较(基于二次型代理)和最坏情况保证,这些方法将极因子的不精确性视为需要"被排除"的干扰因素。我们证明,即使在简单强凸函数如$L(W)=\frac12\|W\|_{\text{F}}^2$上,这些视角已显不足,这表明理解Muon需要超越局部代理和悲观的最坏情况界限。相反,我们的分析揭示了两个在简单二次型上已影响行为且未被主流抽象充分捕捉的观测:(i)极步骤中的近似误差可定性改变离散时间动力学并改善可达性与有限时间性能——这是实践者用于调优Muon的效应,但现有理论主要将其视为纯粹的精度折衷;(ii)目标函数的结构特性会影响有限预算常数,这超出了当前基于条件数的解释范围。因此,任何覆盖这些案例的通用理论必须明确纳入这些要素,或解释为何它们在关注域中无关紧要。