The empirical success of deep learning is often attributed to scaling laws that predict consistent gains as model, data, and compute grow; however, large models can exhibit training instability and diminishing returns, suggesting that scaling laws describe what success looks like but not when and why scaling succeeds or fails. A central obstacle is the lack of a rigorous understanding of feature learning at large depth. While muP characterizes feature-learning dynamics in the infinite-width limit and enables hyperparameter transfer across width, its depth extension (depth-muP) breaks down for residual blocks with more than one internal layer. We derive Neural Feature Dynamics (NFD) for ResNets with single-layer residual blocks, characterizing feature learning via a coupled forward-backward stochastic system in the joint infinite-width and infinite-depth limit. In this regime, NFD identifies when scaling-law trends persist and explains diminishing returns. It also reveals a vanishing mechanism induced by the 1/sqrt(depth) residual scaling under which the gradient-independence assumption (GIA), known to fail during training at finite depth, becomes provably valid again at infinite depth, yielding an analytically tractable regime for end-to-end feature learning. Motivated by this insight, we study two-layer residual blocks and show that the same mechanism causes feature-learning collapse in the first internal layer at large depth, providing a structural explanation for the empirical failure of depth-muP. Based on this diagnosis, we propose a depth-aware learning-rate correction that counteracts the collapse and empirically restores depth-wise hyperparameter transfer, yielding stronger performance in deeper ResNets.

翻译：深度学习的经验成功常归因于尺度定律，该定律预测随着模型、数据和计算资源的增长会带来持续收益；然而，大型模型可能表现出训练不稳定性和收益递减现象，这表明尺度定律描述的是成功时的表现，而非解释尺度扩展何时及为何成功或失败。一个核心障碍在于缺乏对深层特征学习的严格理解。虽然muP在无限宽度极限下刻画了特征学习动力学，并实现了跨宽度的超参数迁移，但其深度扩展（depth-muP）在残差块包含多于一个内部层时会失效。我们针对具有单层残差块的ResNets推导了神经特征动力学（NFD），在联合无限宽度与无限深度极限下，通过一个耦合的前向-后向随机系统刻画了特征学习。在此机制下，NFD识别了尺度定律趋势何时持续存在，并解释了收益递减现象。它还揭示了一种由1/√深度残差缩放诱导的消失机制，在该机制下，已知在有限深度训练中失效的梯度独立性假设（GIA）在无限深度下被证明再次成立，从而为端到端特征学习提供了一个可解析处理的机制。受此启发，我们研究了两层残差块，并表明相同的机制会导致在较大深度时第一个内部层的特征学习崩溃，从而为depth-muP的经验失效提供了结构性解释。基于此诊断，我们提出了一种深度感知的学习率校正方法，以抵消这种崩溃，并在经验上恢复了深度方向的超参数迁移，从而在更深的ResNets中实现了更强的性能。