Deep neural networks (DNNs) exhibit a remarkable ability to automatically learn data representations, finding appropriate features without human input. Here we present a method for analysing feature learning by decomposing DNNs into 1) a forward feature-map $\Phi$ that maps the input dataspace to the post-activations of the penultimate layer, and 2) a final linear layer that classifies the data. We diagonalize $\Phi$ with respect to the gradient descent operator and track feature learning by measuring how the eigenfunctions and eigenvalues of $\Phi$ change during training. Across many popular architectures and classification datasets, we find that DNNs converge, after just a few epochs, to a minimal feature (MF) regime dominated by a number of eigenfunctions equal to the number of classes. This behaviour resembles the neural collapse phenomenon studied at longer training times. For other DNN-data combinations, such as a fully connected network on CIFAR10, we find an extended feature (EF) regime where significantly more features are used. Optimal generalisation performance upon hyperparameter tuning typically coincides with the MF regime, but we also find examples of poor performance within the MF regime. Finally, we recast the phenomenon of neural collapse into a kernel picture which can be extended to broader tasks such as regression.

翻译：深度神经网络（DNNs）展现出自动学习数据表示的卓越能力，能够在无需人工干预的情况下找到合适的特征。本文提出一种分析特征学习的方法，该方法将DNN分解为：1）将输入数据空间映射至倒数第二层激活后状态的前向特征映射 $\Phi$；2）对数据进行分类的最终线性层。我们通过相对于梯度下降算子对角化 $\Phi$，并通过测量 $\Phi$ 的特征函数与特征值在训练过程中的变化来追踪特征学习。在多种主流架构与分类数据集上，我们发现DNN仅需几个训练周期即可收敛至一个由特征函数数量等于类别数所主导的极小特征（MF）区域。此行为类似于在更长训练时间下研究的神经坍缩现象。对于其他DNN-数据组合（例如在CIFAR10数据集上的全连接网络），我们则发现了一个使用显著更多特征的扩展特征（EF）区域。经超参数调优后的最佳泛化性能通常与MF区域重合，但我们也发现了MF区域内性能不佳的实例。最后，我们将神经坍缩现象重新表述为核视角，该视角可扩展至回归等更广泛的任务中。