Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human values. However, RLHF is susceptible to \emph{reward hacking}, where the agent exploits flaws in the reward function rather than learning the intended behavior, thus degrading alignment. Although reward shaping helps stabilize RLHF and partially mitigate reward hacking, a systematic investigation into shaping techniques and their underlying principles remains lacking. To bridge this gap, we present a comprehensive study of the prevalent reward shaping methods. Our analysis suggests two key design principles: (1) the RL reward should be bounded, and (2) the RL reward benefits from rapid initial growth followed by gradual convergence. Guided by these insights, we propose Preference As Reward (PAR), a novel approach that leverages the latent preferences embedded within the reward model as the signal for reinforcement learning. Moreover, PAR exhibits two critical variance-reduction properties that contribute to stabilizing the RLHF training process and effectively extending the tolerance window for early stopping. We evaluated PAR on the base model Gemma2-2B using two datasets, Ultrafeedback-Binarized and HH-RLHF. Experimental results demonstrate PAR's superior performance over other reward shaping methods. On the AlpacaEval 2.0 benchmark, PAR achieves a win rate of at least 5 percentage points higher than competing approaches. Furthermore, PAR exhibits remarkable data efficiency, requiring only a single reference reward for optimal performance, and maintains robustness against reward hacking even after two full epochs of training. The code is available at https://github.com/PorUna-byte/PAR.

翻译：基于人类反馈的强化学习（RLHF）对于使大型语言模型（LLMs）与人类价值观对齐至关重要。然而，RLHF容易受到**奖励攻击**的影响，即智能体利用奖励函数中的缺陷而非学习预期行为，从而损害对齐效果。尽管奖励塑形有助于稳定RLHF并部分缓解奖励攻击，但对于塑形技术及其内在原理的系统性研究仍然缺乏。为填补这一空白，我们对主流的奖励塑形方法进行了全面研究。我们的分析提出了两个关键设计原则：(1) RL奖励应当有界；(2) RL奖励受益于初始阶段的快速增长和后续的逐步收敛。基于这些见解，我们提出了偏好即奖励（PAR），一种新颖的方法，它利用奖励模型中隐含的偏好作为强化学习的信号。此外，PAR展现出两个关键的方差缩减特性，有助于稳定RLHF训练过程，并有效扩展了早停的容忍窗口。我们在基础模型Gemma2-2B上使用Ultrafeedback-Binarized和HH-RLHF两个数据集评估了PAR。实验结果表明，PAR的性能优于其他奖励塑形方法。在AlpacaEval 2.0基准测试中，PAR的胜率比竞争方法至少高出5个百分点。此外，PAR表现出卓越的数据效率，仅需单个参考奖励即可实现最优性能，并且即使在两个完整训练周期后仍能保持对奖励攻击的鲁棒性。代码可在https://github.com/PorUna-byte/PAR获取。