Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual pre-training of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

翻译：语言模型（LMs）对自然语言处理的快速发展起到了关键作用。本文研究语言模型的持续预训练，特别是持续领域自适应预训练（即持续DAP训练）。现有研究表明，使用领域语料对语言模型进行进一步预训练以使其适应特定领域，可以提升该领域的端任务性能。本文提出一种新方法，通过一系列无标注的领域语料对语言模型进行持续DAP训练，使其适应这些领域并提升端任务性能。该方法的核心创新在于一种软掩码机制，可直接控制对语言模型的更新。此外，我们还提出一种新型代理机制，用于保留原始语言模型中的通用知识。该方法通过对比先前学习的领域知识（包括预训练语言模型中的通用知识）与当前完整网络中的知识，实现知识整合。该方法不仅能克服灾难性遗忘，还能实现知识迁移以提升端任务性能。实验评估验证了该方法的有效性。