Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.

翻译：双手灵巧操作因其每只手的高自由度及双手间的协调要求，在机器人学中仍面临重大挑战。现有的单手操作技术常利用人类示范来指导强化学习方法，但难以泛化至涉及多个子技能的复杂双手任务。本文提出VTAO-BiManip，一个结合视觉-触觉-动作预训练与物体理解的新型框架，旨在通过课程式强化学习实现类人的双手操作。我们通过融入手部运动数据改进了先前方法，相比二元触觉反馈，为双手协调提供了更有效的指导。我们的预训练模型利用掩码多模态输入预测未来动作以及物体姿态与尺寸，促进了跨模态正则化。为应对多技能学习挑战，我们引入了一种两阶段课程式强化学习方法以稳定训练过程。我们在瓶盖拧开任务上评估了所提方法，证明了其在仿真和真实环境中的有效性。我们的方法取得了超过现有视觉-触觉预训练方法20%以上的成功率。