Manipulating objects without grasping them is an essential component of human dexterity, referred to as non-prehensile manipulation. Non-prehensile manipulation may enable more complex interactions with the objects, but also presents challenges in reasoning about gripper-object interactions. In this work, we introduce Hybrid Actor-Critic Maps for Manipulation (HACMan), a reinforcement learning approach for 6D non-prehensile manipulation of objects using point cloud observations. HACMan proposes a temporally-abstracted and spatially-grounded object-centric action representation that consists of selecting a contact location from the object point cloud and a set of motion parameters describing how the robot will move after making contact. We modify an existing off-policy RL algorithm to learn in this hybrid discrete-continuous action representation. We evaluate HACMan on a 6D object pose alignment task in both simulation and in the real world. On the hardest version of our task, with randomized initial poses, randomized 6D goals, and diverse object categories, our policy demonstrates strong generalization to unseen object categories without a performance drop, achieving an 89% success rate on unseen objects in simulation and 50% success rate with zero-shot transfer in the real world. Compared to alternative action representations, HACMan achieves a success rate more than three times higher than the best baseline. With zero-shot sim2real transfer, our policy can successfully manipulate unseen objects in the real world for challenging non-planar goals, using dynamic and contact-rich non-prehensile skills. Videos can be found on the project website: https://hacman-2023.github.io.

翻译：非抓取操作是人类灵巧操作的重要组成部分，指不通过抓取而操控物体的能力。这类操作虽能实现更复杂的物体交互，但同时也为推理夹爪-物体相互作用带来了挑战。本文提出混合演员-评论家操作地图（HACMan），一种基于点云观测的6自由度非抓取物体操作强化学习方法。HACMan提出一种时间抽象且空间锚定的以物体为中心的动作表征，该表征包含从物体点云中选择接触位置，以及描述接触后机器人运动方式的一组运动参数。我们改进现有离策略强化学习算法，以适配这种混合离散-连续动作表征的学习。在仿真和真实环境中，我们通过一项6自由度物体位姿对齐任务对HACMan进行评估。在任务最困难版本中（包含随机初始位姿、随机6自由度目标及多样化物体类别），我们的策略展现出对未见物体类别强大的泛化能力且性能不降，在仿真中对未见物体达到89%的成功率，在真实世界零样本迁移中达到50%的成功率。与替代动作表征相比，HACMan的成功率比最佳基线方法高出三倍以上。通过零样本仿真到真实迁移，我们的策略能够成功在真实世界中操控未见物体完成具有挑战性的非平面目标，并运用动态且富含接触的非抓取技能。相关视频可在项目网站查阅：https://hacman-2023.github.io。