From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via offline maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only lose information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a generation-verification gap, (1) it is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.

翻译：从第一性原理的视角来看，基础模型微调（FT）的最强结果是通过一个相对复杂的两阶段训练流程实现的，这似乎有些奇怪。具体而言，研究者首先在某个数据集（例如人类偏好）上训练一个奖励模型（RM），然后将其作为下游强化学习（RL）流程的一部分来提供在线反馈，而不是通过离线最大似然估计直接在该数据集上优化策略参数。事实上，从信息论的视角来看，通过奖励模型传递信息只会导致信息损失，而无法通过在线策略采样创造任何新信息。为了解释这种差异，我们通过理论和实证的视角，仔细审视了关于RL在FT中价值的几种假设。在所考虑的假设中，我们发现最有支持的论点是：在存在生成-验证差距的问题上，（1）从偏好数据中学习相对简单的RM（验证器）是相对容易的。然后，（2）下游RL流程仅返回对此类相对简单的验证器而言最优的策略（生成器）。因此，端到端的两阶段在线FT只需在全策略空间的一个缩减子集中进行搜索，所需数据量少于离线FT。