Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing. Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is known as Neural Collapse ($\mathcal{NC}$). Recent papers have theoretically shown that $\mathcal{NC}$ emerges in the global minimizers of training problems with the simplified ``unconstrained feature model''. In this context, we take a step further and prove the $\mathcal{NC}$ occurrences in deep linear networks for the popular mean squared error (MSE) and cross entropy (CE) losses, showing that global solutions exhibit $\mathcal{NC}$ properties across the linear layers. Furthermore, we extend our study to imbalanced data for MSE loss and present the first geometric analysis of $\mathcal{NC}$ under bias-free setting. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of orthogonal vectors, whose lengths depend on the amount of data in their corresponding classes. Finally, we empirically validate our theoretical analyses on synthetic and practical network architectures with both balanced and imbalanced scenarios.

翻译：现代深度神经网络在图像分类、自然语言处理等任务上取得了令人瞩目的性能。令人惊讶的是，这些拥有海量参数的复杂系统，在训练至收敛时，其最后一层特征和分类器在规范数据集上呈现出相同的结构属性。具体而言，人们观察到最后一层特征坍缩至其类均值，而这些类均值构成一个等角紧框架（ETF）的顶点。这一现象被称为神经坍缩（$\mathcal{NC}$）。近期论文从理论上表明，在采用简化“无约束特征模型”的训练问题的全局最小化中会出现$\mathcal{NC}$。在此基础上，我们更进一步，证明在深度线性网络中，对于常用的均方误差（MSE）和交叉熵（CE）损失函数，全局解在线性层之间展现出$\mathcal{NC}$属性。此外，我们将研究扩展到MSE损失下的非平衡数据，并在无偏置设置下首次给出$\mathcal{NC}$的几何分析。我们的结果表明，最后一层特征和分类器会收敛至由正交向量构成的几何结构，这些向量的长度取决于其对应类别的数据量。最后，我们在平衡与非平衡场景下，通过合成和实际网络架构对理论分析进行实证验证。