Increasing the size of overparameterized neural networks has been a key in achieving state-of-the-art performance. This is captured by the double descent phenomenon, where the test loss follows a decreasing-increasing-decreasing pattern as model width increases. However, the effect of label noise on the test loss curve has not been fully explored. In this work, we uncover an intriguing phenomenon where label noise leads to a \textit{final ascent} in the originally observed double descent curve. Specifically, under a sufficiently large noise-to-sample-size ratio, optimal generalization is achieved at intermediate widths. Through theoretical analysis, we attribute this phenomenon to the shape transition of test loss variance induced by label noise. Furthermore, we extend the final ascent phenomenon to model density and provide the first theoretical characterization showing that reducing density by randomly dropping trainable parameters improves generalization under label noise. We also thoroughly examine the roles of regularization and sample size. Surprisingly, we find that larger $\ell_2$ regularization and robust learning methods against label noise exacerbate the final ascent. We confirm the validity of our findings through extensive experiments on ReLu networks trained on MNIST, ResNets trained on CIFAR-10/100, and InceptionResNet-v2 trained on Stanford Cars with real-world noisy labels.

翻译：增加过参数化神经网络的规模是实现最优性能的关键，这一现象通过双下降曲线得以刻画：随着模型宽度增加，测试损失呈现"下降-上升-下降"的模态。然而，标签噪声对测试损失曲线的影响尚未得到充分探索。本文发现了一个有趣的现象：标签噪声会导致原始双下降曲线出现\textit{最终上升}阶段。具体而言，在噪声与样本量比值足够大时，最优泛化性能在中间宽度处达到。通过理论分析，我们将该现象归因于标签噪声引发的测试损失方差形态转变。进一步地，我们将最终上升现象扩展至模型密度，首次通过理论刻画表明：通过随机丢弃可训练参数降低密度，可在标签噪声下改善泛化性能。我们还深入研究了正则化与样本量的作用。令人惊讶的是，较大的$\ell_2$正则化与针对标签噪声的鲁棒学习方法反而会加剧最终上升现象。我们通过在MNIST上训练的ReLU网络、CIFAR-10/100上训练的ResNet，以及带有真实噪声标签的Stanford Cars上训练的InceptionResNet-v2开展的大量实验，验证了上述发现的可靠性。