Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their ability to follow user instructions well. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following process faces three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are 45x cheaper than crowdworkers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, DPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release all components of AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.

翻译：大型语言模型（LLMs），如ChatGPT，因其出色地遵循用户指令的能力而得到广泛采用。开发这些LLMs涉及一个复杂且理解尚浅的工作流程，需要结合人类反馈进行训练。复现并理解这一指令遵循过程面临三大挑战：数据收集成本高昂、缺乏可信评估以及缺少参考方法实现。我们通过AlpacaFarm应对这些挑战，这是一个能够低成本开展从反馈中学习的研究与开发的仿真器。首先，我们设计了用于模拟人类反馈的LLM提示，其成本比众包工作者低45倍，且与人类判断高度一致。其次，我们提出了一种自动评估方法，并基于真实交互中获取的人类指令对其进行了验证。第三，我们贡献了多种从成对反馈中学习的方法（PPO、DPO、best-of-n、专家迭代等）的参考实现。最后，作为AlpacaFarm的端到端验证，我们在10,000对真实人类反馈上训练并评估了十一个模型，结果显示，在AlpacaFarm中训练的模型排名与基于人类数据训练的模型排名一致。作为AlpacaFarm可能开展的研究示例，我们发现使用奖励模型的方法能够显著优于监督微调，且我们的参考PPO实现相较于Davinci003在胜率上提升了+10%。我们在https://github.com/tatsu-lab/alpaca_farm上发布了AlpacaFarm的所有组件。