Multi-Task Learning (MTL) is widely-accepted in Natural Language Processing as a standard technique for learning multiple related tasks in one model. Training an MTL model requires having the training data for all tasks available at the same time. As systems usually evolve over time, (e.g., to support new functionalities), adding a new task to an existing MTL model usually requires retraining the model from scratch on all the tasks and this can be time-consuming and computationally expensive. Moreover, in some scenarios, the data used to train the original training may be no longer available, for example, due to storage or privacy concerns. In this paper, we approach the problem of incrementally expanding MTL models' capability to solve new tasks over time by distilling the knowledge of an already trained model on n tasks into a new one for solving n+1 tasks. To avoid catastrophic forgetting, we propose to exploit unlabeled data from the same distributions of the old tasks. Our experiments on publicly available benchmarks show that such a technique dramatically benefits the distillation by preserving the already acquired knowledge (i.e., preventing up to 20% performance drops on old tasks) while obtaining good performance on the incrementally added tasks. Further, we also show that our approach is beneficial in practical settings by using data from a leading voice assistant.

翻译：多任务学习（Multi-Task Learning, MTL）在自然语言处理中被广泛接受，作为一种在单个模型中学习多个相关任务的标准技术。训练MTL模型需要同时获取所有任务的训练数据。由于系统通常会随时间演进（例如，为了支持新功能），向现有MTL模型添加新任务通常需要从头开始在所有任务上重新训练模型，这可能耗时且计算成本高昂。此外，在某些场景中，用于原始训练的数据可能不再可用，例如由于存储或隐私问题。本文通过将已训练模型在n个任务上的知识蒸馏到新模型以解决n+1个任务的方式，探讨了逐步扩展MTL模型随时间解决新任务能力的问题。为避免灾难性遗忘，我们提出利用与旧任务同分布的未标注数据。我们在公开基准上的实验表明，该技术通过保留已获取的知识（即在旧任务上防止高达20%的性能下降），同时在新增任务上获得良好的性能，显著有利于知识蒸馏。此外，我们还通过使用来自主流语音助手的数据展示了我们方法在实际设置中的优势。