Foundation models, specifically Large Language Models (LLM's), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.

翻译：基础模型，特别是大型语言模型（LLM），近来获得了广泛关注和应用。基于人类反馈的强化学习（RLHF）涉及训练一个奖励模型以捕捉期望的行为，随后利用该模型对齐LLM。此外，这些奖励模型在推理阶段还被用于评估LLM响应与期望行为的一致性。然而，目前鲜有工作衡量这些奖励模型对分布偏移的鲁棒性。在本研究中，我们通过准确度和校准度（即准确度与置信度之间的一致性）来评估奖励模型性能受分布偏移的影响。我们发现了由超出分布（OOD）的提示和响应导致的新的校准模式与准确度下降，并且奖励模型对响应偏移的敏感性高于对提示偏移的敏感性。此外，我们将一种常用于分类任务的OOD检测技术适配到奖励模型场景中，以检测提示和响应中的这些分布偏移。