Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing. Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is known as Neural Collapse (NC). Recent papers have theoretically shown that NC emerges in the global minimizers of training problems with the simplified "unconstrained feature model". In this context, we take a step further and prove the NC occurrences in deep linear networks for the popular mean squared error (MSE) and cross entropy (CE) losses, showing that global solutions exhibit NC properties across the linear layers. Furthermore, we extend our study to imbalanced data for MSE loss and present the first geometric analysis of NC under bias-free setting. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of orthogonal vectors, whose lengths depend on the amount of data in their corresponding classes. Finally, we empirically validate our theoretical analyses on synthetic and practical network architectures with both balanced and imbalanced scenarios.

翻译：现代深度神经网络在从图像分类到自然语言处理等任务上取得了令人瞩目的性能。令人惊讶的是，这些参数庞大的复杂系统在训练至收敛时，其最后一层特征与分类器在不同标准数据集上展现出相同的结构特性。具体而言，观察到最后一层特征坍缩至其类别均值，且这些类别均值构成等角紧框架（ETF）的顶点。这一现象被称为神经坍缩（NC）。近期理论研究表明，在简化“无约束特征模型”的训练问题全局最小化点中，NC 自然出现。在此基础上，我们进一步证明深度线性网络在常用均方误差（MSE）和交叉熵（CE）损失函数下均存在NC现象，表明全局解在各线性层中均呈现NC特性。此外，我们将研究扩展至MSE损失下的非平衡数据场景，首次实现了无偏置设置下NC的几何分析。结果表明，最后一层特征与分类器收敛至由正交向量构成的几何结构，其长度取决于对应类别的数据量。最后，我们在平衡与非平衡场景下的合成网络与实用网络架构上验证了理论分析的有效性。