Dexterous manipulation is a critical aspect of human capability, enabling interaction with a wide variety of objects. Recent advancements in learning from human demonstrations and teleoperation have enabled progress for robots in such ability. However, these approaches either require complex data collection such as costly human effort for eye-robot contact, or suffer from poor generalization when faced with novel scenarios. To solve both challenges, we propose a framework, DexH2R, that combines human hand motion retargeting with a task-oriented residual action policy, improving task performance by bridging the embodiment gap between human and robotic dexterous hands. Specifically, DexH2R learns the residual policy directly from retargeted primitive actions and task-oriented rewards, eliminating the need for labor-intensive teleoperation systems. Moreover, we incorporate test-time guidance for novel scenarios by taking in desired trajectories of human hands and objects, allowing the dexterous hand to acquire new skills with high generalizability. Extensive experiments in both simulation and real-world environments demonstrate the effectiveness of our work, outperforming prior state-of-the-arts by 40% across various settings.

翻译：灵巧操作是人类能力的关键方面，使其能够与多种物体进行交互。近年来，基于人类演示和遥操作的学习方法取得了进展，使机器人具备了此类能力。然而，这些方法要么需要复杂的数据收集，例如耗费大量人力进行眼-机器人接触，要么在面对新场景时泛化能力较差。为解决这两个挑战，我们提出了一个框架 DexH2R，它将人手运动重定向与面向任务的残差动作策略相结合，通过弥合人类灵巧手与机器人灵巧手之间的具身差距来提高任务性能。具体而言，DexH2R 直接从重定向的原始动作和面向任务的奖励中学习残差策略，无需劳动密集型的遥操作系统。此外，我们通过引入人手和物体的期望轨迹，为新场景融入了测试时引导，使灵巧手能够以高泛化性学习新技能。在仿真和真实环境中的大量实验证明了我们工作的有效性，在各种设置下均优于先前的最优方法 40%。