Continual Learning (CL) aims to learn a sequence of problems (i.e., tasks and domains) by transferring knowledge acquired on previous problems, whilst avoiding forgetting of past ones. Different from previous approaches which focused on CL for one NLP task or domain in a specific use-case, in this paper, we address a more general CL setting to learn from a sequence of problems in a unique framework. Our method, HOP, permits to hop across tasks and domains by addressing the CL problem along three directions: (i) we employ a set of adapters to generalize a large pre-trained model to unseen problems, (ii) we compute high-order moments over the distribution of embedded representations to distinguish independent and correlated statistics across different tasks and domains, (iii) we process this enriched information with auxiliary heads specialized for each end problem. Extensive experimental campaign on 4 NLP applications, 5 benchmarks and 2 CL setups demonstrates the effectiveness of our HOP.

翻译：持续学习旨在通过迁移先前问题（即任务和领域）中获取的知识来学习一系列问题，同时避免遗忘过去的知识。不同于以往针对特定用例下单一NLP任务或领域持续学习的方法，本文在统一框架下处理更通用的持续学习场景。我们提出的HOP方法通过三个方向实现任务与领域的跳跃：（i）采用适配器集合泛化大型预训练模型至未知问题；（ii）计算嵌入表示分布的高阶矩，以区分不同任务与领域间的独立统计量与相关统计量；（iii）通过专为每个终端问题设计的辅助网络头处理增强后的信息。在4个NLP应用、5个基准测试和2种持续学习设置上的广泛实验验证了HOP方法的有效性。