Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their strong instruction-following abilities. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following requires tackling three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are 50x cheaper than crowdworkers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, DPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release all components of AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.

翻译：像ChatGPT这样的大型语言模型（LLMs）因其强大的指令遵循能力而得到广泛应用。然而，开发这些LLMs涉及一个复杂且尚未被充分理解的工作流程，需要通过人类反馈进行训练。要复现并理解这一指令遵循过程，需应对三大挑战：高昂的数据收集成本、缺乏可靠的评估方法，以及缺少基准方法实现。针对这些问题，我们提出AlpacaFarm——一个低成本模拟器，用于支持从反馈中学习的研究与开发。首先，我们设计LLM提示来模拟人类反馈，其成本仅为众包标注员的1/50，且与人类反馈高度一致。其次，我们提出一种自动评估方法，并在真实交互场景中的人类指令上验证了其有效性。第三，我们贡献了多种基于成对反馈学习方法的基准实现（包括PPO、DPO、best-of-n、专家迭代等）。最后，作为AlpacaFarm的端到端验证，我们基于10k对人类真实反馈数据训练并评估了十一个模型，结果表明在AlpacaFarm中训练的模型排名与基于人类数据训练的模型排名一致。作为演示，我们发现使用奖励模型的方法可显著优于监督微调，且我们的参考PPO实现比Davinci003的胜率提升了10%。我们已在https://github.com/tatsu-lab/alpaca_farm 开放AlpacaFarm所有组件。