Spectral bias, the tendency of neural networks to learn low frequencies first, can be both a blessing and a curse. While it enhances the generalization capabilities by suppressing high-frequency noise, it can be a limitation in scientific tasks that require capturing fine-scale structures. The delayed generalization phenomenon known as grokking is another barrier to rapid training of neural networks. Grokking has been hypothesized to arise as learning transitions from the NTK to the feature-rich regime. This paper explores the impact of preconditioned gradient descent (PGD), such as Gauss-Newton, on spectral bias and grokking phenomena. We demonstrate through theoretical and empirical results how PGD can mitigate issues associated with spectral bias. Additionally, building on the rich learning regime grokking hypothesis, we study how PGD can be used to reduce delays associated with grokking. Our conjecture is that PGD, without the impediment of spectral bias, enables uniform exploration of the parameter space in the NTK regime. Our experimental results confirm this prediction, providing strong evidence that grokking represents a transitional behavior between the lazy regime characterized by the NTK and the rich regime. These findings deepen our understanding of the interplay between optimization dynamics, spectral bias, and the phases of neural network learning.

翻译：谱偏置（即神经网络倾向于先学习低频成分的特性）既可能带来益处也可能造成限制。虽然它通过抑制高频噪声增强了泛化能力，但在需要捕捉精细结构的科学任务中可能成为制约因素。被称为"顿悟"的延迟泛化现象是神经网络快速训练的另一个障碍。现有假设认为，顿悟现象产生于学习从神经正切核（NTK）机制向特征丰富机制的过渡过程。本文探讨了预条件梯度下降（如高斯-牛顿法）对谱偏置和顿悟现象的影响。我们通过理论与实证结果论证了预条件梯度下降如何缓解谱偏置相关的问题。此外，基于丰富学习机制的顿悟假说，我们研究了如何利用预条件梯度下降减少顿悟相关的延迟。我们的推测是：预条件梯度下降在不受谱偏置阻碍的情况下，能够在NTK机制中实现对参数空间的均匀探索。实验结果证实了这一预测，为"顿悟现象代表以NTK为特征的惰性机制与丰富机制之间的过渡行为"这一观点提供了有力证据。这些发现深化了我们对优化动力学、谱偏置与神经网络学习阶段之间相互作用的理解。