For a real-world decision-making problem, the reward function often needs to be engineered or learned. A popular approach is to utilize human feedback to learn a reward function for training. The most straightforward way to do so is to ask humans to provide ratings for state-action pairs on an absolute scale and take these ratings as reward samples directly. Another popular way is to ask humans to rank a small set of state-action pairs by preference and learn a reward function from these preference data. Recently, preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. In this work, we develop a theoretical comparison between these human feedback approaches in offline contextual bandits and show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches. Through this, our results seek to provide a theoretical explanation for the empirical successes of preference-based methods from a modeling perspective.

翻译：对于现实世界的决策问题，奖励函数通常需要人工设计或学习。一种常用方法是通过人类反馈学习奖励函数以用于训练。最直接的方法是让人类对状态-动作对按绝对尺度进行评分，并将这些评分直接作为奖励样本。另一种流行方法是让人类根据偏好对少量状态-动作对进行排序，并利用这些偏好数据学习奖励函数。近年来，基于偏好的方法在InstructGPT等实际应用中取得了显著成功。本研究在离线上下文Bandit框架下对这些人类反馈方法进行了理论比较，揭示了人类偏差和反馈建模中的不确定性如何影响这些方法的理论保证。基于此，我们的结果旨在从建模角度为基于偏好方法的经验成功提供理论解释。