The ability to learn manipulation skills by watching videos of humans has the potential to unlock a new source of highly scalable data for robot learning. Here, we tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions. Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors, especially for robots without human-like hands. A promising way forward is to use a modular policy design, leveraging a dedicated grasp generator to produce stable grasps. However, arbitrary stable grasps are often not task-compatible, hindering the robot's ability to perform the desired downstream motion. To address this challenge, we present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data processed by paired grasp-trajectory filtering in simulation. This simulation step extends the trajectory data with grasp suitability labels, which allows for supervised learning of task-oriented grasping capabilities. We show through real-world experiments that our framework can be used to learn precise manipulation skills efficiently without any robot data, resulting in significantly more robust performance than using a grasp generator naively.

翻译：通过观看人类视频学习操作技能的能力，有望为机器人学习开辟新的高可扩展数据源。本文聚焦于抓取操作任务——此类任务要求在完成物体抓取后执行多种抓取后动作。人类视频能为学习抓取后动作提供有效信号，但对于学习前置抓取行为（尤其是缺乏类人灵巧手的机器人）帮助有限。一个有前景的方案是采用模块化策略设计，利用专用抓取生成器产生稳定抓取动作。然而，任意稳定抓取动作往往与任务不兼容，阻碍机器人执行预期的下游动作。针对这一挑战，我们提出"感知-仿真-模仿"(PSI)框架，该框架通过仿真环境中配对抓取-轨迹过滤处理人类视频运动数据，训练模块化操作策略。仿真步骤为轨迹数据添加抓取适用性标签，从而实现对任务导向型抓取能力的监督学习。通过真实世界实验验证，本框架可在无需任何机器人数据的情况下高效学习精准操作技能，较之直接使用抓取生成器的方法，展现出显著更稳健的性能表现。