Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.

翻译：[translated abstract in Chinese] 基于矩阵的优化器（如Muon）能够显著加速语言模型的预训练，但采用标准解耦常数权重衰减时，其相较于AdamW的优势随模型规模和数据量增长而逐渐减弱。我们提出Hyperball——一种解决该问题的简易优化器封装方法。给定基础优化器（如Adam或Muon），Hyperball将权重矩阵的Frobenius范数及其对应优化器更新量均设置为固定常数。在Qwen3系列高达1.2B参数的模型中，Muon Hyperball相比权重衰减基线实现了20%-30%的token等效加速。与解耦权重衰减相比，Hyperball还改善了学习率在不同宽度和深度之间的迁移性能。该方法受此前理论启发，该理论表明：使用权重衰减训练会导致权重范数收敛至仅依赖于训练超参数的平衡态。通过该机制，权重衰减进而决定了角学习率，即权重矩阵方向变化的速率。