Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization~(CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare them with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force

翻译：稀疏奖励下的多目标机器人操作任务对强化学习算法而言具有挑战性，原因在于难以高效收集成功经验。近年提出的后见经验回放（HER）等算法通过利用失败轨迹，将期望目标替换为已实现的状态，使任何失败轨迹均可为学习提供贡献，从而加速学习进程。然而，HER均匀选择失败轨迹，未考虑哪些轨迹对学习最具价值。本文针对此问题，提出一种创新方法——基于接触能量的优先排序（CEBP），利用机器人抓手处的触觉传感器与物体位移信息，依据接触产生的丰富信息从经验回放缓冲区中选择样本。我们的优先排序策略倾向于采样富含接触经验的样本，这类样本通常提供最大信息量。我们在多种稀疏奖励机器人任务上评估所提方法，并与现有最优方法进行对比。结果表明，在机器人操作任务中，我们的方法表现优于或持平于对比方法。最后，我们将训练策略部署至真实Franka机器人执行拾取放置任务，观察到机器人能够成功完成该任务。相关视频与代码已公开于：https://erdiphd.github.io/HER_force