The phenomenon of model-wise double descent, where the test error peaks and then reduces as the model size increases, is an interesting topic that has attracted the attention of researchers due to the striking observed gap between theory and practice \citep{Belkin2018ReconcilingMM}. Additionally, while double descent has been observed in various tasks and architectures, the peak of double descent can sometimes be noticeably absent or diminished, even without explicit regularization, such as weight decay and early stopping. In this paper, we investigate this intriguing phenomenon from the optimization perspective and propose a simple optimization-based explanation for why double descent sometimes occurs weakly or not at all. To the best of our knowledge, we are the first to demonstrate that many disparate factors contributing to model-wise double descent (initialization, normalization, batch size, learning rate, optimization algorithm) are unified from the viewpoint of optimization: model-wise double descent is observed if and only if the optimizer can find a sufficiently low-loss minimum. These factors directly affect the condition number of the optimization problem or the optimizer and thus affect the final minimum found by the optimizer, reducing or increasing the height of the double descent peak. We conduct a series of controlled experiments on random feature models and two-layer neural networks under various optimization settings, demonstrating this optimization-based unified view. Our results suggest the following implication: Double descent is unlikely to be a problem for real-world machine learning setups. Additionally, our results help explain the gap between weak double descent peaks in practice and strong peaks observable in carefully designed setups.

翻译：模型级双重下降现象（即测试误差随模型规模增大先达到峰值后下降）因其理论与实践的显著差异，已成为研究者关注的有趣课题\citep{Belkin2018ReconcilingMM}。值得注意的是，尽管双重下降已在多种任务和架构中被观察到，但其峰值有时会明显减弱甚至消失——即便未采用显式正则化手段（如权重衰减和早停）。本文从优化视角探究这一引人入胜的现象，提出基于优化的简洁解释，阐明为何双重下降有时表现微弱或完全不出现。据我们所知，我们首次证明：多种促成模型级双重下降的异质因素（初始化、归一化、批大小、学习率、优化算法）均可统一于优化视角——当且仅当优化器能找到充分低损失的最小值时，模型级双重下降才会被观察到。这些因素直接影响优化问题或优化器的条件数，进而影响优化器最终找到的最小值，从而降低或抬升双重下降峰值的高度。我们通过随机特征模型与两层神经网络在多种优化设置下的系列控制实验，验证了这种基于优化的统一视角。研究结果表明：双重下降在真实机器学习场景中不太可能成为问题。此外，我们的结果有助于解释实践中观察到的弱双重下降峰值与精心设计场景中可观测的强峰值之间的差异。