Large language models (LLMs) have shown impressive performance in following natural language instructions to solve unseen tasks. However, it remains unclear whether models truly understand task definitions and whether the human-written definitions are optimal. In this paper, we systematically study the role of task definitions in instruction learning. We first conduct an ablation analysis informed by human annotations to understand which parts of a task definition are most important, and find that model performance only drops substantially when removing contents describing the task output, in particular label information. Next, we propose an automatic algorithm to compress task definitions to a minimal supporting set of tokens, and find that 60\% of tokens can be removed while maintaining or even improving model performance. Based on these results, we propose two strategies to help models better leverage task instructions: (1) providing only key information for tasks in a common structured format, and (2) adding a meta-tuning stage to help the model better understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L improvement over 119 unseen test tasks.

翻译：大语言模型（LLMs）在遵循自然语言指令解决未见任务方面展现出令人瞩目的性能。然而，模型是否真正理解任务定义，以及人工编写的定义是否最优，仍有待探究。本文系统研究了任务定义在指令学习中的作用。首先，我们基于人工标注进行消融分析，以理解任务定义中哪些部分最为关键，发现仅当移除描述任务输出（尤其是标签信息）的内容时，模型性能才会显著下降。其次，我们提出一种自动算法将任务定义压缩至最小支持词元集，发现可移除60%的词元而保持甚至提升模型性能。基于这些结果，我们提出两种策略以帮助模型更好地利用任务指令：（1）以通用结构化格式仅为任务提供关键信息，（2）增加元调优阶段以帮助模型更好理解定义。通过这两种策略，我们在119个未见测试任务上实现了4.2个Rouge-L指标的提升。