Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual learning of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

翻译：语言模型（LMs）对自然语言处理的快速发展起到了关键作用。本文研究语言模型的持续学习，特别是持续领域自适应预训练（或持续DAP训练）。已有研究表明，使用领域语料库对LM进行进一步预训练以适应特定领域，能够提升该领域下游任务的表现。本文提出一种新方法，通过一系列无标注领域语料库对LM进行持续DAP训练，使其适应多个领域，从而改进这些领域中的下游任务性能。该方法的核心创新在于一种软掩码机制，可直接控制对LM的更新；同时引入一种新型代理机制，以保留原始LM中的通用知识。此外，该方法通过对比先前学习的领域知识（包括预训练LM中的通用知识）与当前完整网络中的知识，实现知识整合。该方法不仅克服了灾难性遗忘，还能通过知识迁移提升下游任务表现。实验结果验证了所提方法的有效性。