Bimanual manipulation is a longstanding challenge in robotics due to the large number of degrees of freedom and the strict spatial and temporal synchronization required to generate meaningful behavior. Humans learn bimanual manipulation skills by watching other humans and by refining their abilities through play. In this work, we aim to enable robots to learn bimanual manipulation behaviors from human video demonstrations and fine-tune them through interaction. Inspired by seminal work in psychology and biomechanics, we propose modeling the interaction between two hands as a serial kinematic linkage -- as a screw motion, in particular, that we use to define a new action space for bimanual manipulation: screw actions. We introduce ScrewMimic, a framework that leverages this novel action representation to facilitate learning from human demonstration and self-supervised policy fine-tuning. Our experiments demonstrate that ScrewMimic is able to learn several complex bimanual behaviors from a single human video demonstration, and that it outperforms baselines that interpret demonstrations and fine-tune directly in the original space of motion of both arms. For more information and video results, https://robin-lab.cs.utexas.edu/ScrewMimic/

翻译：双臂操作由于自由度众多且需要严格的空间与时间同步以产生有意义的动作，一直是机器人学中的长期挑战。人类通过观察他人学习双臂操作技能，并通过实践完善自身能力。本研究旨在使机器人能够从人类视频演示中学习双臂操作行为，并通过交互进行微调。受心理学和生物力学开创性工作的启发，我们提出将双手间的交互建模为串联运动链——特别是螺旋运动，据此定义了一个新的双臂操作动作空间：螺旋动作。我们提出了ScrewMimic框架，利用这种新颖的动作表征促进从人类演示中学习及自我监督策略微调。实验表明，ScrewMimic能够从单段人类视频演示中学习多种复杂双臂行为，并且优于在原始双臂运动空间中直接解释演示并微调的基线方法。更多信息及视频结果请见：https://robin-lab.cs.utexas.edu/ScrewMimic/