Foundation models, specifically Large Language Models (LLM's), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align an LLM. These reward models are additionally used at inference-time to estimate how well LLM responses adhere to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting in order to detect these distribution shifts in prompts and responses.

翻译：基础模型，特别是大型语言模型（LLM），近年来获得了广泛的关注和应用。基于人类反馈的强化学习（RLHF）涉及训练一个奖励模型来捕捉期望行为，随后用于对齐大型语言模型。这些奖励模型还在推理时用于估计语言模型输出对期望行为的遵循程度。然而，目前鲜有研究衡量这些奖励模型对分布偏移的鲁棒性。在本工作中，我们评估了奖励模型性能——通过准确性和校准度（即准确性与置信度之间的一致性）来衡量——如何受到分布偏移的影响。我们展示了由分布外（OOD）提示和响应导致的新型校准模式与准确性下降，并表明奖励模型对响应偏移的敏感性高于提示偏移。此外，我们将分类任务中常用的分布外检测技术适配到奖励模型场景中，以检测提示和响应中的这些分布偏移。