Human-in-the-loop guidance has emerged as an effective approach for enabling faster convergence in online reinforcement learning (RL) of complex real-world manipulation tasks. However, existing human-in-the-loop RL (HiL-RL) frameworks often suffer from low sample efficiency, requiring substantial human interventions to achieve convergence and thereby leading to high labor costs. To address this, we propose a sample-efficient real-world human-in-the-loop RL framework named \method, which requires fewer human intervention by actively selecting informative samples. Specifically, stable reduction of policy entropy enables improved trade-off between exploration and exploitation with higher sample efficiency. We first build influence functions of different samples on the policy entropy, which is efficiently estimated by the covariance of action probabilities and soft advantages of policies. Then we select samples with moderate values of influence functions, where shortcut samples that induce sharp entropy drops and noisy samples with negligible effect are pruned. Extensive experiments on four real-world manipulation tasks demonstrate that \method achieves a 42.1\% higher success rate while requiring 10.1\% fewer human interventions compared to the state-of-the-art HiL-RL method, validating its effectiveness. The project page providing code, videos, and mathematical formulations can be found at https://e2hil.github.io/.

翻译：人机协同指导已成为在复杂现实世界操作任务的在线强化学习（RL）中实现更快收敛的有效方法。然而，现有的人机协同强化学习（HiL-RL）框架通常存在样本效率低下的问题，需要大量人工干预才能实现收敛，从而导致高昂的人力成本。为解决这一问题，我们提出了一种名为 \method 的样本高效现实世界人机协同强化学习框架，该框架通过主动选择信息丰富的样本来减少所需的人工干预。具体而言，策略熵的稳定降低能够以更高的样本效率改进探索与利用之间的权衡。我们首先构建了不同样本对策略熵的影响函数，该函数通过策略的动作概率与软优势的协方差进行高效估计。随后，我们选择影响函数值适中的样本，同时剔除那些导致熵急剧下降的捷径样本以及影响可忽略的噪声样本。在四个现实世界操作任务上进行的大量实验表明，与最先进的 HiL-RL 方法相比，\method 在减少 10.1\% 人工干预的同时，实现了 42.1\% 更高的成功率，验证了其有效性。提供代码、视频和数学公式的项目页面可在 https://e2hil.github.io/ 找到。