Reinforcement learning from human feedback (RLHF) has become an important technical and storytelling tool to deploy the latest machine learning systems. In this book, we hope to give a gentle introduction to the core methods for people with some level of quantitative background. The book starts with the origins of RLHF -- both in recent literature and in a convergence of disparate fields of science in economics, philosophy, and optimal control. We then set the stage with definitions, problem formulation, data collection, and other common math used in the literature. The core of the book details every optimization stage in using RLHF, from starting with instruction tuning to training a reward model and finally all of rejection sampling, reinforcement learning, and direct alignment algorithms. The book concludes with advanced topics -- understudied research questions in synthetic data and evaluation -- and open questions for the field.

翻译：基于人类反馈的强化学习（RLHF）已成为部署最新机器学习系统的重要技术手段与叙事工具。本书旨在为具备一定数理背景的读者提供关于该领域核心方法的简明导论。开篇追溯RLHF的学术渊源——既涵盖近期文献，也涉及经济学、哲学与最优控制等多学科领域的交汇脉络。随后通过定义阐释、问题建模、数据采集及文献中常用数学工具的介绍奠定理论基础。本书核心章节系统解析RLHF各优化阶段：从指令微调入门，到奖励模型训练，最终涵盖拒绝采样、强化学习及直接对齐算法等完整技术链条。末篇探讨前沿议题——包括合成数据与评估机制等尚未充分探索的研究方向——以及该领域亟待解决的开放性问题。