Giving up and starting over may seem wasteful in many situations such as searching for a target or training deep neural networks (DNNs). Our study, though, demonstrates that resetting from a checkpoint can significantly improve generalization performance when training DNNs with noisy labels. In the presence of noisy labels, DNNs initially learn the general patterns of the data but then gradually memorize the corrupted data, leading to overfitting. By deconstructing the dynamics of stochastic gradient descent (SGD), we identify the behavior of a latent gradient bias induced by noisy labels, which harms generalization. To mitigate this negative effect, we apply the stochastic resetting method to SGD, inspired by recent developments in the field of statistical physics achieving efficient target searches. We first theoretically identify the conditions where resetting becomes beneficial, and then we empirically validate our theory, confirming the significant improvements achieved by resetting. We further demonstrate that our method is both easy to implement and compatible with other methods for handling noisy labels. Additionally, this work offers insights into the learning dynamics of DNNs from an interpretability perspective, expanding the potential to analyze training methods through the lens of statistical physics.

翻译：在许多情境下，例如搜索目标或训练深度神经网络（DNNs），放弃并重新开始看似是一种浪费。然而，我们的研究表明，在使用带噪声标签训练DNNs时，从检查点进行重置可以显著提升泛化性能。在存在噪声标签的情况下，DNNs最初会学习数据的一般模式，但随后会逐渐记忆被污染的数据，从而导致过拟合。通过解构随机梯度下降（SGD）的动态过程，我们识别出由噪声标签引起的潜在梯度偏差行为，这种偏差损害了泛化能力。为了缓解这种负面影响，我们受统计物理学领域近期在实现高效目标搜索方面进展的启发，将随机重置方法应用于SGD。我们首先从理论上确定了重置变得有益的条件，随后通过实验验证了我们的理论，确认了重置带来的显著改进。我们进一步证明，我们的方法不仅易于实现，而且与其他处理噪声标签的方法兼容。此外，这项工作从可解释性的视角为理解DNNs的学习动态提供了见解，拓展了通过统计物理学视角分析训练方法的潜力。