Fine-tuning has been proven to be a simple and effective technique to transfer the learned knowledge of Pre-trained Language Models (PLMs) to downstream tasks. However, vanilla fine-tuning easily overfits the target data and degrades the generalization ability. Most existing studies attribute it to catastrophic forgetting, and they retain the pre-trained knowledge indiscriminately without identifying what knowledge is transferable. Motivated by this, we frame fine-tuning into a causal graph and discover that the crux of catastrophic forgetting lies in the missing causal effects from the pretrained data. Based on the causal view, we propose a unified objective for fine-tuning to retrieve the causality back. Intriguingly, the unified objective can be seen as the sum of the vanilla fine-tuning objective, which learns new knowledge from target data, and the causal objective, which preserves old knowledge from PLMs. Therefore, our method is flexible and can mitigate negative transfer while preserving knowledge. Since endowing models with commonsense is a long-standing challenge, we implement our method on commonsense QA with a proposed heuristic estimation to verify its effectiveness. In the experiments, our method outperforms state-of-the-art fine-tuning methods on all six commonsense QA datasets and can be implemented as a plug-in module to inflate the performance of existing QA models.

翻译：微调已被证明是一种简单有效的技术，可将预训练语言模型（PLMs）学到的知识迁移至下游任务。然而，原始微调方法容易过拟合目标数据，导致泛化能力下降。现有研究多将其归因于灾难性遗忘，但它们在保留预训练知识时未区分哪些知识是可迁移的。受此启发，我们将微调过程构建为因果图，发现灾难性遗忘的关键在于预训练数据缺失因果效应。基于因果视角，我们提出一种统一的微调目标以恢复因果性。有趣的是，该统一目标可视为原始微调目标（从目标数据学习新知识）与因果目标（保留PLMs的旧知识）之和。因此，我们的方法灵活且能在保留知识的同时缓解负迁移。鉴于赋予模型常识是一个长期挑战，我们在常识问答任务上应用该方法，并采用启发式估计验证其有效性。实验结果表明，我们的方法在六个常识问答数据集上均优于现有最优微调方法，且可作为即插即用模块提升现有问答模型的性能。