Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/

翻译：行为克隆已成为机器人学习的高效范式。然而，行为克隆在采集示范数据后缺乏在线改进的自引导机制。现有的离线到在线学习方法常因离线数据与在线学习之间的分布不匹配，导致策略用新学的不良动作替代先前学习的优秀动作。本文提出Q2RL算法（从行为克隆中提取Q估计与Q门控用于强化学习），这是一种高效的离线到在线学习方法。该方法包含两部分：（1）Q估计——通过与环境的少量交互步骤从行为克隆策略中提取Q函数，随后进行在线强化学习；（2）Q门控——根据行为克隆策略与强化学习策略动作各自的Q值进行切换，为强化学习策略训练采集样本。在D4RL和robomimic基准的操控任务中，Q2RL在成功率和收敛时间上均优于最先进的离线到在线学习基线。Q2RL效率足以应用于机器人实机强化学习场景，可在1-2小时的在线交互中为管道组装和零件装配等高精度接触操控任务学习鲁棒策略，最高实现100%的成功率，较原始行为克隆策略提升高达3.75倍。代码与视频见https://pages.rai-inst.com/q2rl_website/