Incremental Learning (IL) has been a long-standing problem in both vision and Natural Language Processing (NLP) communities. In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP. Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue. However, we find that this assumption is problematic. Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (Class-Incremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs. Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs. The results show that SEQ* has competitive or superior performance compared to state-of-the-art (SOTA) IL methods and requires considerably less trainable parameters and training time. These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs. The data, code and scripts are publicly available at https://github.com/zzz47zzz/pretrained-lm-for-incremental-learning.

翻译：增量学习（IL）一直是视觉和自然语言处理（NLP）领域的长期难题。近年来，随着预训练语言模型（PLMs）在各类NLP下游任务中取得显著进展，将其作为骨干网络已成为NLP增量学习研究中的常见做法。多数研究认为灾难性遗忘是实现优秀增量学习性能的最大障碍，并提出了各种技术来克服这一问题。然而，我们发现这一假设存在问题。具体而言，我们在两种最流行的增量学习设置（类增量学习和任务增量学习）下，对四个分类任务（文本分类、意图分类、关系抽取和命名实体识别）中的20余种方法进行了重新审视，结果揭示这些方法大多严重低估了PLMs固有的抗遗忘能力。基于这一观察，我们提出了一种名为SEQ*的极其简单的增量学习方法。实验结果表明，与最先进的增量学习方法相比，SEQ*具备相当或更优的性能，且所需的可训练参数数量和训练时间显著更少。这些发现促使我们重新审视基于PLMs的增量学习，并鼓励未来研究从根本上理解PLMs中的灾难性遗忘现象。相关数据、代码和脚本已在https://github.com/zzz47zzz/pretrained-lm-for-incremental-learning 公开提供。