Robot policies need to adapt to human preferences and/or new environments. Human experts may have the domain knowledge required to help robots achieve this adaptation. However, existing works often require costly offline re-training on human feedback, and those feedback usually need to be frequent and too complex for the humans to reliably provide. To avoid placing undue burden on human experts and allow quick adaptation in critical real-world situations, we propose designing and sparingly presenting easy-to-answer pairwise action preference queries in an online fashion. Our approach designs queries and determines when to present them to maximize the expected value derived from the queries' information. We demonstrate our approach with experiments in simulation, human user studies, and real robot experiments. In these settings, our approach outperforms baseline techniques while presenting fewer queries to human experts. Experiment videos, code and appendices are found at https://sites.google.com/view/onlineactivepreferences.

翻译：机器人策略需要适应人类偏好和/或新环境。人类专家可能拥有帮助机器人实现这种适应所需的领域知识。然而，现有工作通常需要基于人类反馈进行代价高昂的离线再训练，且这些反馈往往需要频繁提供，且对人类而言过于复杂而难以可靠给出。为避免给人类专家带来过度负担，并允许在关键的现实场景中快速适应，我们提出以在线方式设计并审慎呈现易于回答的成对动作偏好查询。我们的方法设计查询并决定何时呈现这些查询，以最大化从查询信息中获得的期望价值。我们通过仿真实验、人类用户研究和真实机器人实验展示了该方法。在这些场景中，我们的方法在向人类专家呈现更少查询的同时，优于基线技术。实验视频、代码及附录参见 https://sites.google.com/view/onlineactivepreferences。