We formalize and study a phenomenon called feature collapse that makes precise the intuitive idea that entities playing a similar role in a learning task receive similar representations. As feature collapse requires a notion of task, we leverage a simple but prototypical NLP task to study it. We start by showing experimentally that feature collapse goes hand in hand with generalization. We then prove that, in the large sample limit, distinct words that play identical roles in this NLP task receive identical local feature representations in a neural network. This analysis reveals the crucial role that normalization mechanisms, such as LayerNorm, play in feature collapse and in generalization.

翻译：我们形式化并研究了一种名为“特征坍塌”的现象，该现象精确描述了学习任务中扮演相似角色的实体获得相似表征的直观概念。由于特征坍塌需要任务概念作为支撑，我们借助一项简单但典型的自然语言处理任务进行研究。首先通过实验证明，特征坍塌与泛化能力密切相关。接着我们证明，在大样本极限条件下，该任务中扮演相同角色的不同词汇在神经网络中将获得完全相同的局部特征表征。该分析揭示了LayerNorm等归一化机制在特征坍塌与泛化过程中所起的关键作用。