Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

翻译：基于偏好的强化学习提供了一个框架，通过人类对两种行为的偏好来训练智能体。然而，基于偏好的强化学习在规模化方面面临挑战，因为它需要大量人类反馈来学习与人类意图对齐的奖励函数。本文提出偏好Transformer——一种利用Transformer架构对人类偏好进行建模的神经结构。与以往假设人类判断基于同等贡献的马尔可夫奖励的方法不同，我们引入了一种基于非马尔可夫奖励加权和的新型偏好模型。随后，我们利用堆叠因果和双向自注意力层的Transformer架构设计了所提出的偏好模型。我们证明，偏好Transformer能够利用真实人类反馈解决多种控制任务，而以往方法无法成功。我们还表明，偏好Transformer通过自动捕捉人类决策过程中的时间依赖性，可以推导出规范明确的奖励函数并关注轨迹中的关键事件。代码已发布在项目网站：https://sites.google.com/view/preference-transformer。