While reinforcement learning (RL) enables robots to acquire skills autonomously, its real-world deployment is severely limited by inefficient and unsafe exploration. Human-in-the-loop interventions offer a practical solution, yet existing methods typically exploit these interventions as auxiliary training signals, without fully capturing the richer information they provide about when and how autonomy should be guided. Human interventions often encode relative preferences over behavior under safety and task constraints, rather than prescribing exact actions to imitate. Motivated by this perspective, we propose Online Human Preference as Guidance in Reinforcement Learning (OHP-RL), a framework that leverages human interventions as preference information to guide policy learning. OHP-RL introduces a state-dependent preference gate that adaptively regulates when and to what extent human interventions should shape policy learning. This design enables the agent to benefit from intermittent and imperfect human feedback while preserving autonomous exploration and stable policy optimization. We evaluate OHP-RL on three challenging real-world contact-rich manipulation tasks on a Franka robot. Across all tasks, OHP-RL consistently achieves strong success rates, faster convergence, and substantially lower human intervention effort than prior approaches. Moreover, the learned policies exhibit more stable and human-aligned behavior throughout training.

翻译：尽管强化学习使机器人能够自主获取技能，但其在实际部署中因低效且不安全的探索而严重受限。人类在环干预提供了一种实用的解决方案，但现有方法通常仅将这些干预用作辅助训练信号，未能充分捕捉它们所蕴含的关于何时以及如何引导自主性的丰富信息。人类的干预往往编码了在安全与任务约束下对行为的相对偏好，而非规定精确的动作进行模仿。受此观点启发，我们提出了一种名为OHP-RL（基于人类在线偏好的强化学习引导）的框架，该框架利用人类干预作为偏好信息来指导策略学习。OHP-RL引入了一个依赖于状态的偏好门控机制，自适应地调节人类干预应在何时及多大程度上影响策略学习。这一设计使智能体能够从非连续且不完美的人类反馈中受益，同时保持自主探索与稳定的策略优化。我们在Franka机器人上针对三项具有挑战性的真实世界接触密集型操作任务评估了OHP-RL。在所有任务中，OHP-RL均稳定实现了较高的成功率、更快的收敛速度，且相较现有方法大幅降低了人类干预成本。此外，学习到的策略在整个训练过程中展现出更稳定且符合人类预期的行为模式。