Real-world reinforcement learning (RL) offers a promising approach to training precise and dexterous robotic manipulation policies in an online manner, enabling robots to learn from their own experience while gradually reducing human labor. However, prior real-world RL methods often assume that human interventions are optimal across the entire state space, overlooking the fact that even expert operators cannot consistently provide optimal actions in all states or completely avoid mistakes. Indiscriminately mixing intervention data with robot-collected data inherits the sample inefficiency of RL, while purely imitating intervention data can ultimately degrade the final performance achievable by RL. The question of how to leverage potentially suboptimal and noisy human interventions to accelerate learning without being constrained by them thus remains open. To address this challenge, we propose SiLRI, a state-wise Lagrangian reinforcement learning algorithm for real-world robot manipulation tasks. Specifically, we formulate the online manipulation problem as a constrained RL optimization, where the constraint bound at each state is determined by the uncertainty of human interventions. We then introduce a state-wise Lagrange multiplier and solve the problem via a min-max optimization, jointly optimizing the policy and the Lagrange multiplier to reach a saddle point. Built upon a human-as-copilot teleoperation system, our algorithm is evaluated through real-world experiments on diverse manipulation tasks. Experimental results show that SiLRI effectively exploits human suboptimal interventions, reducing the time required to reach a 90% success rate by at least 50% compared with the state-of-the-art RL method HIL-SERL, and achieving a 100% success rate on long-horizon manipulation tasks where other RL methods struggle to succeed. Project website: https://silri-rl.github.io/.

翻译：现实世界强化学习（RL）为在线训练精确灵巧的机器人操作策略提供了一种前景广阔的方法，使机器人能够从自身经验中学习，同时逐步减少人力投入。然而，现有的现实世界RL方法通常假设人类干预在整个状态空间内都是最优的，忽略了即使专家操作者也无法在所有状态下始终提供最优动作或完全避免错误的事实。不加区分地将干预数据与机器人收集的数据混合，会继承RL的样本低效性问题；而单纯模仿干预数据则可能最终限制RL可达到的性能上限。因此，如何利用可能次优且含噪声的人类干预来加速学习，同时又不被其限制，仍是一个开放性问题。为应对这一挑战，我们提出了SiLRI，一种面向现实世界机器人操作任务的状态拉格朗日强化学习算法。具体而言，我们将在线操作问题建模为带约束的RL优化问题，其中每个状态的约束边界由人类干预的不确定性决定。随后，我们引入状态拉格朗日乘子，并通过最小-最大优化求解该问题，联合优化策略与拉格朗日乘子以达到鞍点。基于人机协同遥操作系统，我们的算法在多种操作任务上进行了现实世界实验验证。实验结果表明，SiLRI能有效利用人类次优干预，与最先进的RL方法HIL-SERL相比，达到90%成功率所需时间至少缩短50%，并在其他RL方法难以完成的长时序操作任务上实现了100%的成功率。项目网站：https://silri-rl.github.io/。