Instruction tuning has shown its ability to not only enhance zero-shot generalization across various tasks but also its effectiveness in improving the performance of specific tasks. A crucial aspect in instruction tuning for a particular task is a strategic selection of related tasks that offer meaningful supervision, thereby enhancing efficiency and preventing performance degradation from irrelevant tasks. Our research reveals that leveraging instruction information \textit{alone} enables the identification of pertinent tasks for instruction tuning. This approach is notably simpler compared to traditional methods that necessitate complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Furthermore, by additionally learning the unique instructional template style of the meta-dataset, we observe an improvement in task selection accuracy, which contributes to enhanced overall performance. Experimental results demonstrate that training on a small set of tasks, chosen solely based on the instructions, leads to substantial performance improvements on benchmarks like P3, Big-Bench, NIV2, and Big-Bench Hard. Significantly, these improvements exceed those achieved by prior task selection methods, highlighting the efficacy of our approach.

翻译：指令微调不仅能够提升模型在多种任务上的零样本泛化能力，还能有效改善特定任务的性能。针对特定任务进行指令微调时，关键环节在于策略性地选择能够提供有意义监督的相关任务，从而提升效率并避免因无关任务导致的性能下降。本研究表明，单独利用指令信息即可识别指令微调中的相关任务。相较于传统方法——这类方法需要测量任务间复杂的成对可迁移性或为目标任务创建数据样本——本方法显著更为简单。此外，通过额外学习元数据集的独特指令模板风格，我们观察到任务选择准确率有所提升，进而增强了整体性能。实验结果表明，仅基于指令选择的少量任务进行训练，可在P3、Big-Bench、NIV2和Big-Bench Hard等基准测试中实现显著的性能提升。值得注意的是，这些改进超过了以往任务选择方法所取得的效果，凸显了我们方法的有效性。