Human manipulation videos are a convenient and intuitive source for robot learning. However, directly transferring human dexterity to robots remains challenging due to perception errors and embodiment gap. To address this, we introduce Video2Sim2Real, a full-stack framework for autonomous skill acquisition from a single human manipulation video. Our framework first uses off-the-shelf foundation models to reconstruct a simulator-ready digital twin and extract robot and object motion priors. Rather than treating the extracted robot motion as a reliable reference throughout execution, our key idea is to recover and leverage the most fundamental sources of supervision from the demonstrated skill: We identify object-centric keyframes to optimize the corresponding robot configurations using object information from the simulator, and use these configurations as anchors that refine the robot motion such that it ultimately has the desired impact on the environment. To bridge the remaining sim-to-real gap, we introduce a sim-to-real strategy that decouples robustness to noisy and incomplete perception from variations in hand-object interaction dynamics. Specifically, we learn to recalibrate robot configurations from noisy real-world point clouds via IL, and leverage residual RL to perform local finger-level adaptations to ensure for robust and effective interactions. Finally, a collision-aware motion planning module enables spatial generalization to novel object configurations. Across several everyday manipulation tasks, Video2Sim2Real improves simulated task success, safety, and trajectory coherence over numerous baselines, and achieves better sim-to-real transfer than existing techniques. These results demonstrate a promising path toward autonomous dexterous skill acquisition from human videos.

翻译：人类操作视频为机器人学习提供了便捷且直观的途径。然而，由于感知误差和具身鸿沟的存在，直接将人类灵巧性迁移至机器人仍具挑战。为此，我们提出Video2Sim2Real——一种从单段人类操作视频中实现自主技能获取的全栈式框架。该框架首先利用现成的基础模型重建可适配模拟器的数字孪生体，并提取机器人与物体的运动先验。我们的核心思想并非将提取的机器人运动视为整个执行过程中的可靠参照，而是恢复并利用示教技能中最基础的监督来源：通过识别以物体为中心的关键帧，利用模拟器中的物体信息优化对应机器人构型，并将这些构型作为锚点来精炼机器人运动，使其最终对环境产生预期影响。为弥合模拟到现实的剩余差距，我们提出一种解耦策略：将感知噪声与不完整性的鲁棒性、以及手-物交互动力学变化分离开来。具体而言，我们通过模仿学习从真实世界含噪点云中重新校准机器人构型，并借助残差强化学习进行局部手指级自适应调整，以确保鲁棒且有效的交互。最终，碰撞感知的运动规划模块实现了对新型物体构型的空间泛化。在多项日常操作任务中，Video2Sim2Real相比基线方法提升了模拟任务成功率、安全性及轨迹一致性，且其模拟到现实的迁移效果优于现有技术。这些成果展示了从人类视频实现自主灵巧技能获取的可行路径。