The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments, a type of reinforcement learning from human feedback (RLHF). These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling human preferences instead as informed by each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences, and we prove that the previous partial return model lacks this identifiability property in multiple contexts. We empirically show that our proposed regret preference model outperforms the partial return preference model with finite training data in otherwise the same setting. Additionally, we find that our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research. We have open sourced our experimental code, the human preferences dataset we gathered, and our training and preference elicitation interfaces for gathering a such a dataset.

翻译：强化学习的效用受到奖励函数与人类利益相关者利益一致性的限制。一种有前景的对齐方法是从人类对轨迹片段对的偏好中学习奖励函数，这属于基于人类反馈的强化学习（RLHF）范畴。这些人类偏好通常被认为仅由部分回报（即每个片段中奖励之和）决定。我们发现这一假设存在缺陷，并提出将人类偏好建模为基于每个片段的遗憾值（衡量片段与最优决策偏差的指标）。我们证明：在根据遗憾生成无限偏好数据的情况下，可以识别出等价于生成这些偏好的奖励函数，而先前基于部分回报的模型在多种情境下缺乏这种可识别性。实验表明，在相同设定下使用有限训练数据时，我们提出的遗憾偏好模型优于部分回报偏好模型。此外，遗憾偏好模型能更好地预测真实人类偏好，并从中学习到更符合人类意图的奖励函数及策略。总体而言，本研究确立了偏好模型选择的重要性，所提出的遗憾偏好模型改进了近期研究的核心假设。我们已开源实验代码、收集的人类偏好数据集，以及用于采集此类数据的训练与偏好收集界面。