High-quality supervised fine-tuning (SFT) data are crucial for eliciting strong capabilities from pretrained large language models (LLMs). Typically, instructions are paired with multiple responses sampled from other LLMs, which are often out of the distribution of the target model to be fine-tuned. This, at scale, can lead to diminishing returns and even hurt the models' performance and robustness. We propose **GRAPE**, a novel SFT framework that accounts for the unique characteristics of the target model. For each instruction, it gathers responses from various LLMs and selects the one with the highest probability measured by the target model, indicating that it aligns most closely with the target model's pretrained distribution; it then proceeds with standard SFT training. We first evaluate GRAPE with a controlled experiment, where we sample various solutions for each question in UltraInteract from multiple models and fine-tune commonly used LMs like LLaMA3.1-8B, Mistral-7B, and Qwen2.5-7B on GRAPE-selected data. GRAPE significantly outperforms strong baselines, including distilling from the strongest model with an absolute gain of up to 13.8%, averaged across benchmarks, and training on 3x more data with a maximum performance improvement of 17.3%. GRAPE's strong performance generalizes to realistic settings. We experiment with the post-training data used for Tulu3 and Olmo-2. GRAPE outperforms strong baselines trained on 4.5 times more data by 6.1% and a state-of-the-art data selection approach by 3% on average performance. Remarkably, using 1/3 of the data and half the number of epochs, GRAPE enables LLaMA3.1-8B to surpass the performance of Tulu3-SFT by 3.5%.

翻译：高质量的监督微调（SFT）数据对于从预训练大语言模型（LLM）中激发强大能力至关重要。通常，指令会与从其他LLM采样的多个回答配对，而这些回答往往不在待微调目标模型的分布范围内。在规模化应用时，这可能导致收益递减，甚至损害模型的性能和鲁棒性。我们提出了**GRAPE**，一种新颖的SFT框架，该框架考虑了目标模型的独特特性。对于每条指令，它从不同LLM收集回答，并选择由目标模型测得的概率最高的那个，这表明该回答与目标模型的预训练分布最为契合；随后进行标准的SFT训练。我们首先通过对照实验评估GRAPE：从多个模型为UltraInteract中的每个问题采样多种解决方案，并在GRAPE选择的数据上对LLaMA3.1-8B、Mistral-7B和Qwen2.5-7B等常用LM进行微调。GRAPE显著优于多个强基线，包括从最强模型进行蒸馏的方法（在基准测试上平均绝对增益高达13.8%），以及使用3倍数据量训练的方法（最大性能提升达17.3%）。GRAPE的强大性能可推广到实际场景。我们在Tulu3和Olmo-2使用的后训练数据上进行了实验。GRAPE在平均性能上，优于使用4.5倍数据量训练的强基线6.1%，并优于最先进的数据选择方法3%。值得注意的是，仅使用1/3的数据和一半的训练轮数，GRAPE使LLaMA3.1-8B的性能超越了Tulu3-SFT 3.5%。