Learning from Preferences in Reinforcement Learning (PbRL) has gained attention recently, as it serves as a natural fit for complicated tasks where the reward function is not easily available. However, preferences often come with uncertainty and noise if they are not from perfect teachers. Much prior literature aimed to detect noise, but with limited types of noise and most being uniformly distributed with no connection to observations. In this work, we formalize the notion of targeted feature-dependent noise and propose several variants like trajectory feature noise, trajectory similarity noise, uncertainty-aware noise, and Language Model noise. We evaluate feature-dependent noise, where noise is correlated with certain features in complex continuous control tasks from DMControl and Meta-world. Our experiments show that in some feature-dependent noise settings, the state-of-the-art noise-robust PbRL method's learning performance is significantly deteriorated, while PbRL method with no explicit denoising can surprisingly outperform noise-robust PbRL in majority settings. We also find language model's noise exhibits similar characteristics to feature-dependent noise, thereby simulating realistic humans and call for further study in learning with feature-dependent noise robustly.

翻译：基于偏好的强化学习（PbRL）因其适用于奖励函数不易获取的复杂任务而受到关注。然而，若非来自完美教师，偏好数据常伴随不确定性与噪声。现有研究多致力于噪声检测，但所涉噪声类型有限，且大多为与观测无关的均匀分布噪声。本研究形式化了目标特征相关噪声的概念，提出了轨迹特征噪声、轨迹相似度噪声、不确定性感知噪声及语言模型噪声等多种变体。我们在DMControl和Meta-world的复杂连续控制任务中评估了噪声与特定特征相关的特征相关噪声。实验表明，在某些特征相关噪声场景下，当前最先进的抗噪PbRL方法学习性能显著下降，而无显式去噪的PbRL方法在多数场景中反而优于抗噪PbRL。我们还发现语言模型产生的噪声具有与特征相关噪声相似的特征，可模拟真实人类行为，因此需要进一步研究对特征相关噪声具有鲁棒性的学习方法。