Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization. In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain the information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse.

翻译：对预训练语言模型学习到的上下文表示进行微调仍是NLP领域的一种常见做法。然而，微调可能导致表示退化（也称为表示坍塌），进而引发不稳定性、次优性能及泛化能力弱化等问题。本文提出表示投影不变性（REPINA），一种新颖的正则化方法，通过抑制表示中非期望变化，在微调过程中维持表示的信息含量并减少表示坍塌。我们研究了该正则化方法在13项语言理解任务（GLUE基准测试及六个额外数据集）上对比5个可比基线的实证表现。在领域内性能评估中，REPINA在大多数任务（13项中的10项）上持续优于其他基线。我们还展示了其在少样本场景下的有效性及对标签扰动的鲁棒性。此外，作为衍生成果，我们扩展了先前关于表示坍塌的研究，并提出若干量化指标。实证结果表明，我们的方法在缓解表示坍塌方面显著更有效。