Reward functions are difficult to design and often hard to align with human intent. Preference-based Reinforcement Learning (RL) algorithms address these problems by learning reward functions from human feedback. However, the majority of preference-based RL methods na\"ively combine supervised reward models with off-the-shelf RL algorithms. Contemporary approaches have sought to improve performance and query complexity by using larger and more complex reward architectures such as transformers. Instead of using highly complex architectures, we develop a new and parameter-efficient algorithm, Inverse Preference Learning (IPL), specifically designed for learning from offline preference data. Our key insight is that for a fixed policy, the $Q$-function encodes all information about the reward function, effectively making them interchangeable. Using this insight, we completely eliminate the need for a learned reward function. Our resulting algorithm is simpler and more parameter-efficient. Across a suite of continuous control and robotics benchmarks, IPL attains competitive performance compared to more complex approaches that leverage transformer-based and non-Markovian reward functions while having fewer algorithmic hyperparameters and learned network parameters. Our code is publicly released.

翻译：奖励函数难以设计且往往难以与人类意图对齐。基于偏好的强化学习算法通过从人类反馈中学习奖励函数来解决这些问题，然而，大多数基于偏好的强化学习方法简单地将监督式奖励模型与现成的强化学习算法相结合。当代方法尝试通过使用更大、更复杂的奖励架构（例如Transformer）来提升性能和查询复杂度。我们并未采用高度复杂的架构，而是开发了一种新颖且参数高效的算法——逆向偏好学习，该算法专为从离线偏好数据中学习而设计。我们的关键洞察在于：对于固定策略，Q函数编码了奖励函数的所有信息，从而有效实现了二者的可互换性。基于这一洞察，我们完全消除了学习奖励函数的必要性。由此产生的算法更为简洁且参数效率更高。在一系列连续控制与机器人基准测试中，IPL取得了与采用基于Transformer及非马尔可夫奖励函数的复杂方法相竞争的性表现，同时其算法超参数和网络学习参数更少。我们的代码已公开。