In reinforcement learning, specifying reward functions that capture the intended task can be very challenging. Reward learning aims to address this issue by learning the reward function. However, a learned reward model may have a low error on the training distribution, and yet subsequently produce a policy with large regret. We say that such a reward model has an error-regret mismatch. The main source of an error-regret mismatch is the distributional shift that commonly occurs during policy optimization. In this paper, we mathematically show that a sufficiently low expected test error of the reward model guarantees low worst-case regret, but that for any fixed expected test error, there exist realistic data distributions that allow for error-regret mismatch to occur. We then show that similar problems persist even when using policy regularization techniques, commonly employed in methods such as RLHF. Our theoretical results highlight the importance of developing new ways to measure the quality of learned reward models.

翻译：在强化学习中，设计能准确捕捉预期任务的奖励函数往往极具挑战性。奖励学习旨在通过学习奖励函数来解决这一问题。然而，学习得到的奖励模型可能在训练分布上具有较低的误差，却随后产生具有高遗憾度的策略。我们将此类奖励模型称为存在误差-遗憾失配。误差-遗憾失配的主要来源是策略优化过程中普遍存在的分布偏移。本文通过数学证明表明：奖励模型的期望测试误差足够低时，可保证最坏情况下的遗憾度较低；但对于任意固定的期望测试误差，均存在现实的数据分布可能导致误差-遗憾失配的发生。我们进一步证明，即使采用RLHF等方法中常用的策略正则化技术，类似问题依然存在。我们的理论结果凸显了开发衡量学习奖励模型质量新方法的重要性。