Recent work in NLP has shown promising results in training models on large amounts of tasks to achieve better generalization. However, it is not well-understood how tasks are related, and how helpful training tasks can be chosen for a new task. In this work, we investigate whether knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. We provide TaskWeb, a large-scale benchmark of pairwise task transfers for 22 NLP tasks using three different model types, sizes, and adaptation methods, spanning about 25,000 experiments. Then, we design a new method TaskShop based on our analysis of TaskWeb. TaskShop uses TaskWeb to estimate the benefit of using a source task for learning a new target task, and to choose a subset of helpful training tasks for multi-task training. Our method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. We also use TaskShop to build much smaller multi-task training sets that improve zero-shot performances across 11 different target tasks by at least 4.3%.

翻译：近期NLP研究表明，通过大量任务训练模型可提升泛化性能，但任务间的关联性以及如何为新任务选择有效的训练任务尚未得到充分理解。本研究通过成对任务迁移探究任务关系，是否能更优地选择一个或多个源任务以辅助新目标任务的学习。我们构建了TaskWeb——一个覆盖22项NLP任务的成对任务迁移大规模基准，涉及三种不同模型类型、规模及适配方法，总计约25,000组实验。基于TaskWeb分析，我们设计了新方法TaskShop：利用TaskWeb预估源任务对目标任务学习的增益，并选择有助益的训练任务子集进行多任务训练。该方法将源任务的整体排序精度和top-k准确率分别提升10%和38%。同时，我们利用TaskShop构建了规模更小的多任务训练集，使11项不同目标任务的零样本性能至少提升4.3%。