Generating diverse and sophisticated instructions for downstream tasks by Large Language Models (LLMs) is pivotal for advancing the effect. Current approaches leverage closed-source LLMs, employing in-context prompting for instruction generation. However, in this paper, we found that in-context prompting cannot generate complex instructions with length $\ge 100$ for tasks like code completion. To solve this problem, we introduce Ada-Instruct, an adaptive instruction generator developed by fine-tuning open-source LLMs. Our pivotal finding illustrates that fine-tuning open-source LLMs with a mere ten samples generates long instructions that maintain distributional consistency for complex reasoning tasks. We empirically validated Ada-Instruct's efficacy across different applications, including code completion, mathematical reasoning, and commonsense reasoning. The results underscore Ada-Instruct's superiority, evidencing its improvements over its base models, current self-instruct methods, and other state-of-the-art models.

翻译：通过大型语言模型（LLMs）为下游任务生成多样且复杂的指令对推进效果至关重要。现有方法依赖闭源LLMs，利用上下文提示生成指令。然而，本文发现上下文提示无法为代码补全等任务生成长度≥100的复杂指令。为解决此问题，我们提出Ada-Instruct——一种通过微调开源LLMs开发的自适应指令生成器。关键发现表明：仅用十个样本微调开源LLMs即可生成长指令，且这些指令能为复杂推理任务保持分布一致性。我们通过代码补全、数学推理和常识推理等不同应用场景实证验证了Ada-Instruct的有效性。实验结果凸显了Ada-Instruct的优越性，证明其相较于基座模型、现有自指令方法及其他最先进模型均有所改进。