Transferring the optimal learning rate from small to large neural networks can enable efficient training at scales where hyperparameter tuning is otherwise prohibitively expensive. To this end, the Maximal Update Parameterization (muP) proposes a learning rate scaling designed to keep the update dynamics of internal representations stable across different model widths. However, the scaling rules of muP rely on strong assumptions, particularly about the geometric alignment of a layer's inputs with both its weights and gradient updates. In this large-scale empirical investigation, we show that these assumptions hold only briefly at the start of training in the practical setups where learning rate transfer is most valuable, such as LLM training. For the remainder of training it is weight decay rather than muP that correctly stabilizes the update dynamics of internal representations across widths, facilitating learning rate transfer. This suggests muP's scaling primarily acts as a form of implicit learning rate warmup, allowing us to largely replace it with modified warmup schedules. Together these findings fundamentally challenge prevailing beliefs about learning rate transfer and can explain empirical observations such as why muP requires the independent weight decay variant for good transfer.

翻译：将最优学习率从小规模神经网络迁移至大规模网络，能够在超参数调优成本极高的尺度上实现高效训练。为此，最大更新参数化（muP）提出了一种学习率缩放策略，旨在保持内部表征的更新动态在不同模型宽度间的稳定性。然而，muP的缩放规则依赖于强假设，特别是关于层输入与其权重及梯度更新的几何对齐关系。在这项大规模实证研究中，我们证明这些假设仅在训练初期短暂成立——而这恰恰是学习率迁移最具价值的实际场景（例如大语言模型训练）。在后续训练过程中，真正能够稳定不同宽度间内部表征更新动态、从而促进学习率迁移的是权重衰减而非muP。这表明muP的缩放主要起到一种隐式学习率预热的作用，使得我们能够通过改进的预热方案在很大程度上替代它。这些发现共同对当前关于学习率迁移的主流认知提出了根本性质疑，并能解释诸如为何muP需要独立权重衰减变体才能实现良好迁移等实证观察。