Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual learning of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

翻译：语言模型（LMs）在自然语言处理的快速发展中发挥了关键作用。本文研究语言模型的持续学习，特别是持续领域自适应预训练（或持续DAP训练）。现有研究表明，使用领域语料库对语言模型进行进一步预训练以使其适应特定领域，可以提升该领域最终任务的性能。本文提出了一种新方法，通过一系列无标签领域语料库对语言模型进行持续DAP训练，使其适应这些领域，从而提升最终任务性能。该方法的核心创新在于一种软掩蔽机制，可直接控制语言模型的更新。同时，我们提出了一种新的代理机制，以保留原始语言模型中的通用知识。此外，该方法通过对比先前学习的领域知识（包括预训练语言模型中的通用知识）与当前完整网络中的知识表示，实现知识整合。该方法不仅克服了灾难性遗忘问题，还能通过知识迁移提升最终任务的性能。实验评估证明了所提方法的有效性。