Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

翻译：通过监督微调（SFT）利用人工标注数据的力量对于推进大型语言模型（LLM）的发展至关重要。本文深入探讨了在无需额外获取人工标注数据的情况下，如何从弱模型培育出强LLM的可能性。我们提出了一种名为自我博弈微调（SPIN）的新型微调方法，该方法从监督微调模型开始。SPIN的核心在于自我博弈机制，即LLM通过与自身的实例对抗来优化自身能力。具体而言，LLM利用其先前迭代生成的训练数据，通过区分这些自我生成响应与人工标注数据中的响应来优化其策略。我们的方法逐步将LLM从初期模型提升为强大模型，充分挖掘了SFT中人工标注演示数据的潜力。理论上，我们证明了只有当LLM策略与目标数据分布一致时，该方法训练目标函数的全局最优解才能实现。在实证方面，我们在多个基准数据集（包括HuggingFace Open LLM排行榜、MT-Bench以及Big-Bench数据集）上评估了该方法。结果表明，SPIN能显著提升LLM在各类基准测试中的性能，甚至超越那些通过直接偏好优化（DPO）结合额外GPT-4偏好数据训练的模型。这揭示了自我博弈的前景，使LLM无需专家对手即可达到人类水平性能。代码已公开于https://github.com/uclaml/SPIN。