Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing. Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is known as Neural Collapse ($\mathcal{NC}$). Recent papers have theoretically shown that $\mathcal{NC}$ emerges in the global minimizers of training problems with the simplified ``unconstrained feature model''. In this context, we take a step further and prove the $\mathcal{NC}$ occurrences in deep linear networks for the popular mean squared error (MSE) and cross entropy (CE) losses, showing that global solutions exhibit $\mathcal{NC}$ properties across the linear layers. Furthermore, we extend our study to imbalanced data for MSE loss and present the first geometric analysis of $\mathcal{NC}$ under bias-free setting. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of orthogonal vectors, whose lengths depend on the amount of data in their corresponding classes. Finally, we empirically validate our theoretical analyses on synthetic and practical network architectures with both balanced and imbalanced scenarios.

翻译：现代深度神经网络在图像分类到自然语言处理等任务上取得了令人瞩目的性能。令人惊讶的是，这些包含大量参数的复杂系统在训练至收敛时，其最后一层特征和分类器在规范数据集上展现出相同的结构特性。特别地，观察到最后一层特征坍缩到其类别均值，而这些类别均值是等角紧框架（ETF）单纯形的顶点。这一现象被称为神经坍缩（$\mathcal{NC}$）。近期论文已从理论上证明，在简化的“无约束特征模型”训练问题的全局最小化中会出现$\mathcal{NC}$。在此基础上，我们更进一步，证明了在深度线性网络中，对于常用的均方误差（MSE）和交叉熵（CE）损失，全局解跨越线性层展现出$\mathcal{NC}$特性。此外，我们将研究扩展到MSE损失下的非平衡数据，并首次在无偏置设置下对$\mathcal{NC}$进行了几何分析。我们的结果表明，最后一层特征和分类器收敛到由正交向量构成的几何结构，这些向量的长度取决于其对应类别的数据量。最后，我们在平衡与非平衡场景下的合成网络和实际网络架构上，实证验证了我们的理论分析。