Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization. In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain the information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse.

翻译：对预训练语言模型学习到的上下文化表示进行微调仍然是自然语言处理中的常见做法。然而，微调可能导致表示退化（也称为表示坍缩），进而引发不稳定性、次优性能以及弱泛化能力。本文提出一种新型正则化方法——表示投影不变性（Representation Projection Invariance, REPINA），旨在通过抑制表示中的不良变化，在微调过程中维持表示的信息含量并减少表示坍缩。我们研究了所提正则化方法在13项语言理解任务（GLUE基准测试及六个额外数据集）中与5个可比基线的实证性能对比。在领域内性能评估中，REPINA在大多数任务（13项中的10项）上始终优于其他基线。我们还展示了其在少样本场景及标签扰动鲁棒性中的有效性。此外，作为副产品，我们扩展了先前关于表示坍缩的研究，并提出了若干量化指标。实证结果表明，我们的方法在缓解表示坍缩方面显著更为有效。