Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. However, how to select new tasks to improve the performance and generalizability of IT models remains an open question. Training on all existing tasks is impractical due to prohibiting computation requirements, and randomly selecting tasks can lead to suboptimal performance. In this work, we propose active instruction tuning based on prompt uncertainty, a novel framework to identify informative tasks, and then actively tune the models on the selected tasks. We represent the informativeness of new tasks with the disagreement of the current model outputs over perturbed prompts. Our experiments on NIV2 and Self-Instruct datasets demonstrate that our method consistently outperforms other baseline strategies for task selection, achieving better out-of-distribution generalization with fewer training tasks. Additionally, we introduce a task map that categorizes and diagnoses tasks based on prompt uncertainty and prediction probability. We discover that training on ambiguous (prompt-uncertain) tasks improves generalization while training on difficult (prompt-certain and low-probability) tasks offers no benefit, underscoring the importance of task selection for instruction tuning.

翻译：指令微调通过在大规模多样化任务上结合指令训练大语言模型，实现了令人瞩目的零样本泛化效果。然而，如何选择新任务以提升指令微调模型的性能与泛化能力仍是一个开放性问题。由于计算资源限制，对所有现有任务进行训练不切实际，而随机选择任务可能导致次优性能。本文提出基于提示不确定性的主动指令微调——一种识别信息性任务的新框架，并主动对选定任务进行模型微调。我们通过当前模型在扰动提示上的输出分歧来表征新任务的信息量。在NIV2与Self-Instruct数据集上的实验表明，该方法在任务选择策略上持续优于其他基线方法，以更少的训练任务实现了更优的分布外泛化性能。此外，我们提出基于提示不确定性与预测概率的任务图谱，用于任务分类与诊断。研究发现，训练模糊性任务可提升泛化能力，而训练确定性低概率任务则无益，这凸显了任务选择对指令微调的重要性。