Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual learning of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

翻译：语言模型（LMs）在自然语言处理的快速发展中发挥了关键作用。本文研究语言模型的持续学习，特别是持续领域自适应预训练（或称持续DAP训练）。已有研究表明，使用领域语料库对语言模型进行进一步预训练以使其适应特定领域，能够提升该领域的下游任务性能。本文提出了一种新颖方法，通过一系列无标注领域语料库持续对语言模型进行DAP训练，使其适应这些领域，从而提升下游任务性能。该方法的核心创新在于一种软掩蔽机制，能够直接控制对语言模型的更新。同时，本文提出了一种新的代理项以保留原始语言模型中的通用知识。此外，该方法通过对比先前学习的领域知识（包括预训练语言模型中的通用知识）与当前完整网络的知识，实现知识整合。该方法不仅克服了灾难性遗忘，还实现了知识迁移以提升下游任务性能。实验评估证明了所提方法的有效性。