The training dynamics of deep neural networks often defy expectations, even as these models form the foundation of modern machine learning. Two prominent examples are grokking, where test performance improves abruptly long after the training loss has plateaued, and the information bottleneck principle, where models progressively discard input information irrelevant to the prediction task as training proceeds. However, the mechanisms underlying these phenomena and their relations remain poorly understood. In this work, we present a unified explanation of such late-phase phenomena through the lens of neural collapse, which characterizes the geometry of learned representations. We show that the contraction of population within-class variance is a key factor underlying both grokking and information bottleneck, and relate this measure to the neural collapse measure defined on the training set. By analyzing the dynamics of neural collapse, we show that distinct time scales between fitting the training set and the progression of neural collapse account for the behavior of the late-phase phenomena. Finally, we validate our theoretical findings on multiple datasets and architectures.

翻译：深度神经网络的训练动态常常出人意料，尽管这些模型构成了现代机器学习的基础。其中两个典型现象是"顿悟"（grokking）——即在训练损失早已趋于稳定后，测试性能突然显著提升；以及信息瓶颈原理——即模型在训练过程中逐步丢弃与预测任务无关的输入信息。然而，这些现象的内在机制及其相互关系仍不甚明晰。本研究通过神经坍缩的视角——该理论描述了学习表征的几何特性——为这类后期训练现象提供了统一解释。我们证明类内方差的收缩是顿悟与信息瓶颈现象的共同关键因素，并将该度量与训练集上定义的神经坍缩度量建立关联。通过分析神经坍缩的动态过程，我们发现训练集拟合与神经坍缩进程之间的不同时间尺度共同决定了后期训练现象的行为特征。最后，我们在多个数据集和架构上验证了理论发现。