Modern deep neural networks have achieved impressive performance on tasks from image classification to natural language processing. Surprisingly, these complex systems with massive amounts of parameters exhibit the same structural properties in their last-layer features and classifiers across canonical datasets when training until convergence. In particular, it has been observed that the last-layer features collapse to their class-means, and those class-means are the vertices of a simplex Equiangular Tight Frame (ETF). This phenomenon is known as Neural Collapse ($\mathcal{NC}$). Recent papers have theoretically shown that $\mathcal{NC}$ emerges in the global minimizers of training problems with the simplified ``unconstrained feature model''. In this context, we take a step further and prove the $\mathcal{NC}$ occurrences in deep linear networks for the popular mean squared error (MSE) and cross entropy (CE) losses, showing that global solutions exhibit $\mathcal{NC}$ properties across the linear layers. Furthermore, we extend our study to imbalanced data for MSE loss and present the first geometric analysis of $\mathcal{NC}$ under bias-free setting. Our results demonstrate the convergence of the last-layer features and classifiers to a geometry consisting of orthogonal vectors, whose lengths depend on the amount of data in their corresponding classes. Finally, we empirically validate our theoretical analyses on synthetic and practical network architectures with both balanced and imbalanced scenarios.

翻译：现代深度神经网络在从图像分类到自然语言处理的任务中取得了令人瞩目的性能。令人惊讶的是，这些包含大量参数的复杂系统，在训练至收敛时，其最后一层特征和分类器在典型数据集上展现出相同的结构特性。具体而言，已观察到最后一层特征坍缩至其类均值，而这些类均值是等角紧框架（ETF）单形的顶点。这一现象被称为神经坍缩（$\mathcal{NC}$）。近期论文从理论上表明，在简化的“无约束特征模型”下，$\mathcal{NC}$出现在训练问题的全局最小化器中。在此背景下，我们向前迈进一步，证明了在深度线性网络中，对于常用的均方误差（MSE）和交叉熵（CE）损失函数，$\mathcal{NC}$现象会发生，表明全局解在线性层间展现出$\mathcal{NC}$特性。此外，我们将研究扩展到MSE损失下的非平衡数据，并首次在无偏置设定下对$\mathcal{NC}$进行了几何分析。我们的结果展示了最后一层特征和分类器收敛到由正交向量构成的几何结构，这些向量的长度取决于其对应类别的数据量。最后，我们在合成和实际网络架构上，针对平衡和非平衡场景，通过实验验证了我们的理论分析。