Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.

翻译：现代神经网络严重过度参数化，而剪枝（移除冗余神经元或连接）已成为在保持性能的同时压缩网络的关键方法。然而，尽管实际剪枝方法已相当成熟，但剪枝是否会在神经网络中引发尖锐的相变，如果会，它们属于何种普适类，仍是有待解答的问题。为此，我们研究在MNIST数据集上训练的全连接神经网络，通过独立改变训练和评估阶段的丢弃（即移除神经元）率来绘制相图。我们识别出三种截然不同的相态：智能相（网络能够学习）、失忆相（网络已遗忘）和痴呆相（网络无法学习），这些相态通过交叉熵损失随训练数据集规模的幂律标度行为被清晰区分。{在智能相中，损失的代数衰减（机器学习文献中称为神经标度律）从统计力学视角看是准长程有序的标志。}我们证明，智能相与失忆相之间的转变伴随标度不变性，其发散的尺度展现出类似Berezinskii-Kosterlitz-Thouless相变的特征；该相结构在不同网络宽度和深度下均保持稳健。我们的研究结果确立了丢弃诱导的剪枝为通过统计力学视角理解神经网络行为提供了具体场景。