Preference-based Reinforcement Learning (PbRL) enables policy learning through simple queries comparing trajectories from a single policy. While human responses to these queries make it possible to learn policies aligned with human preferences, PbRL suffers from low query efficiency, as policy bias limits trajectory diversity and reduces the number of discriminable queries available for learning preferences. This paper identifies preference discriminability, which quantifies how easily a human can judge which trajectory is closer to their ideal behavior, as a key metric for improving query efficiency. To address this, we move beyond comparisons within a single policy and instead generate queries by comparing trajectories from multiple policies, as training them from scratch promotes diversity without policy bias. We propose Discriminability-Aware Policy-to-Policy Preference-Based Efficient Reinforcement Learning (DAPPER), which integrates preference discriminability with trajectory diversification achieved by multiple policies. DAPPER trains new policies from scratch after each reward update and employs a discriminator that learns to estimate preference discriminability, enabling the prioritized sampling of more discriminable queries. During training, it jointly maximizes the preference reward and preference discriminability score, encouraging the discovery of highly rewarding and easily distinguishable policies. Experiments in simulated and real-world legged robot environments demonstrate that DAPPER outperforms previous methods in query efficiency, particularly under challenging preference discriminability conditions. A supplementary video that facilitates understanding of the proposed framework and its experimental results is available at: https://youtu.be/lRwX8FNN8n4

翻译：基于偏好的强化学习（PbRL）通过比较单一策略产生的轨迹的简单查询来实现策略学习。尽管人类对这些查询的响应使得学习符合人类偏好的策略成为可能，但PbRL存在查询效率低下的问题，因为策略偏差限制了轨迹的多样性，并减少了可用于学习偏好的可区分查询数量。本文提出偏好判别性——即量化人类判断哪条轨迹更接近其理想行为的难易程度——作为提高查询效率的关键指标。为解决此问题，我们超越了单一策略内的比较，转而通过比较多个策略产生的轨迹生成查询，因为从头训练这些策略能在无策略偏差的情况下促进多样性。我们提出了具有判别性感知的策略间偏好高效强化学习（DAPPER），该方法将偏好判别性与通过多策略实现的轨迹多样化相结合。DAPPER在每次奖励更新后从头训练新策略，并采用一个学习估计偏好判别性的判别器，从而实现对更具判别性查询的优先采样。在训练过程中，它同时最大化偏好奖励和偏好判别性分数，鼓励发现高奖励且易于区分的策略。在模拟和现实足式机器人环境中的实验表明，DAPPER在查询效率上优于先前方法，尤其在具有挑战性的偏好判别性条件下。一个有助于理解所提框架及其实验结果的补充视频可在以下网址获取：https://youtu.be/lRwX8FNN8n4