Reward models are key to language model post-training and inference pipelines. Conveniently, recent work showed that every language model defines an implicit reward model (IM-RM), without requiring any architectural changes. However, such IM-RMs tend to generalize worse, especially out-of-distribution, compared to explicit reward models (EX-RMs) that apply a dedicated linear head over the hidden representations of a language model. The existence of a generalization gap is puzzling, as EX-RMs and IM-RMs are nearly identical. They can be trained using the same data, loss function, and language model, and differ only in how the reward is computed. Toward a fundamental understanding of the implicit biases underlying different reward model types, we investigate the root cause of this gap. Our main finding, backed by theory and experiments, is that IM-RMs rely more heavily on superficial token-level cues. Consequently, they often generalize worse than EX-RMs under token-level distribution shifts, as well as in-distribution. Furthermore, we provide evidence against alternative hypotheses for the generalization gap. Most notably, we challenge the claim that IM-RMs struggle in tasks where generation is harder than verification because they can operate both as a verifier and a generator. Overall, our results highlight that seemingly minor design choices can substantially impact the generalization behavior of reward models.

翻译：奖励模型是语言模型后训练与推理流程的关键。近期研究便利地表明，每个语言模型都定义了一个隐式奖励模型（IM-RM），无需任何架构改动。然而，与在语言模型隐藏表示上应用专用线性头的显式奖励模型（EX-RM）相比，此类IM-RM的泛化能力往往更差，尤其在分布外场景中。泛化差距的存在令人困惑，因为EX-RM与IM-RM几乎完全相同：它们可使用相同数据、损失函数和语言模型进行训练，仅奖励计算方式存在差异。为从根本上理解不同奖励模型类型背后的隐式偏差，我们探究了这一差距的根本成因。通过理论与实验验证，我们的主要发现是：IM-RM更依赖于表面的词元级线索。因此，在词元级分布偏移及分布内场景中，其泛化能力通常弱于EX-RM。此外，我们提供了反驳泛化差距其他假设的证据。最值得注意的是，我们质疑了“IM-RM在生成任务比验证任务更困难时表现不佳，因其可同时作为验证器和生成器运作”的论断。总体而言，我们的研究结果表明，看似细微的设计选择可能显著影响奖励模型的泛化行为。