Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task instead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.

翻译：近期，针对多个任务进行指令微调的语言模型（LM），也称为多任务提示微调（MT），已展现出对未见任务的泛化能力。先前研究表明，增加训练任务数量是构建更强大多任务提示微调语言模型的关键因素。在本工作中，我们报告了一项意外发现：仅针对单一任务微调的专家语言模型，在11个不同未见数据集和BIG-bench基准的13个数据集上，分别以平均3.20%和1.29%的准确率优势，超越了基于300余个不同任务训练的多任务提示微调语言模型。这一发现对先前认为仅通过增加任务数量即可增强多任务提示微调语言模型的观点提出了质疑。基于此发现，我们进一步证明，针对每个训练任务分别训练独立的专家语言模型（而非单一多任务提示微调语言模型）进行零样本推理，具有多重优势，包括：（1）避免指令微调中常见的负向任务迁移；（2）能够持续学习新任务而无需重新训练旧任务以规避灾难性遗忘；（3）在合并独立专家模型时展现出组合能力。代码已开源至https://github.com/joeljang/ELM。