Recent work in NLP has shown promising results in training models on large amounts of tasks to achieve better generalization. However, it is not well-understood how tasks are related, and how helpful training tasks can be chosen for a new task. In this work, we investigate whether knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. We provide TaskWeb, a large-scale benchmark of pairwise task transfers for 22 NLP tasks using three different model types, sizes, and adaptation methods, spanning about 25,000 experiments. Then, we design a new method TaskShop based on our analysis of TaskWeb. TaskShop uses TaskWeb to estimate the benefit of using a source task for learning a new target, and to choose a subset of helpful training tasks for multi-task learning. Our method improves overall rankings and top-k precision of source tasks by 12% and 29%, respectively. We also use TaskShop to build smaller multi-task training sets that improve zero-shot performances across 11 different target tasks by at least 4.3%.

翻译：近期NLP领域研究表明，在大量任务上训练模型能够取得更好的泛化效果。然而，任务之间的关联性以及如何为新任务选取有效的训练任务尚不明确。本研究探究了通过成对任务迁移理解任务关联性，能否提升为学习新目标任务而选取一个或多个源任务的效果。我们构建了TaskWeb基准数据集，涵盖22项NLP任务在三种不同模型类型、规模及适配方法下的成对任务迁移，总计约25,000组实验。基于对TaskWeb的分析，我们设计了新方法TaskShop：该方法利用TaskWeb估算使用某源任务学习新目标任务的收益，并为多任务学习选取有效的训练任务子集。所提方法在源任务的总体排序和top-k精确度上分别提升了12%和29%。此外，通过TaskShop构建的较小规模多任务训练集，使得11个不同目标任务在零样本场景下的性能至少提升4.3%。