We introduce Instruct-SkillMix, an automated approach for creating diverse, high quality SFT data. The Instruct-SkillMix pipeline involves two stages, each leveraging an existing powerful LLM: (1) Skill extraction: uses the LLM to extract core "skills" for instruction-following, either from existing datasets, or by directly prompting the model; (2) Data generation: uses the powerful LLM to generate (instruction, response) data that exhibit a randomly chosen pair of these skills. Here, the use of random skill combinations promotes diversity and difficulty. Vanilla SFT (i.e., no PPO, DPO, or RL methods) on data generated from Instruct-SkillMix leads to strong gains on instruction following benchmarks such as AlpacaEval 2.0, MT-Bench, and WildBench. With just $4$K examples, LLaMA-3-8B-Base achieves 42.76% length-controlled win rate on AlpacaEval 2.0. To our knowledge, this achieves state-of-the-art performance among all models that have only undergone SFT (no RL methods) and competes with proprietary models such as Claude 3 Opus and LLaMA-3.1-405B-Instruct. Ablation studies also suggest plausible reasons for why creating open instruction-tuning datasets via naive crowd-sourcing has proved difficult. Introducing low quality answers ("shirkers") in $20\%$ of Instruct-SkillMix examples causes performance to plummet, sometimes catastrophically. The Instruct-SkillMix pipeline is flexible and is adaptable to other settings.

翻译：我们介绍了Instruct-SkillMix，一种用于创建多样化、高质量监督微调（SFT）数据的自动化方法。Instruct-SkillMix流程包含两个阶段，每个阶段都利用现有的强大大型语言模型（LLM）：（1）技能提取：使用LLM从现有数据集中或通过直接提示模型，提取指令遵循的核心“技能”；（2）数据生成：使用强大的LLM生成（指令，响应）数据，这些数据展现出随机选择的一对这些技能。在此，随机技能组合的使用促进了数据的多样性和难度。仅使用Instruct-SkillMix生成的数据进行标准SFT（即不使用近端策略优化PPO、直接偏好优化DPO或强化学习方法），即可在指令遵循基准测试（如AlpacaEval 2.0、MT-Bench和WildBench）上带来显著提升。仅使用$4$K个示例，LLaMA-3-8B-Base模型在AlpacaEval 2.0上就达到了42.76%的长度控制胜率。据我们所知，这在所有仅经过SFT（未使用RL方法）的模型中达到了最先进的性能，并与Claude 3 Opus和LLaMA-3.1-405B-Instruct等专有模型相竞争。消融研究也揭示了为何通过简单的众包方式创建开放的指令微调数据集被证明是困难的。在$20\%$的Instruct-SkillMix示例中引入低质量答案（“逃避者”）会导致性能急剧下降，有时甚至是灾难性的。Instruct-SkillMix流程具有灵活性，可适应其他应用场景。