Future robots are envisioned as versatile systems capable of performing a variety of household tasks. The big question remains, how can we bridge the embodiment gap while minimizing physical robot learning, which fundamentally does not scale well. We argue that learning from in-the-wild human videos offers a promising solution for robotic manipulation tasks, as vast amounts of relevant data already exist on the internet. In this work, we present VidBot, a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos. VidBot leverages a pipeline to extract explicit representations from them, namely 3D hand trajectories from videos, combining a depth foundation model with structure-from-motion techniques to reconstruct temporally consistent, metric-scale 3D affordance representations agnostic to embodiments. We introduce a coarse-to-fine affordance learning model that first identifies coarse actions from the pixel space and then generates fine-grained interaction trajectories with a diffusion model, conditioned on coarse actions and guided by test-time constraints for context-aware interaction planning, enabling substantial generalization to novel scenes and embodiments. Extensive experiments demonstrate the efficacy of VidBot, which significantly outperforms counterparts across 13 manipulation tasks in zero-shot settings and can be seamlessly deployed across robot systems in real-world environments. VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.

翻译：未来的机器人被设想为能够执行各种家庭任务的通用系统。一个关键问题仍然存在：如何在最小化物理机器人学习的同时弥合具身鸿沟，而物理机器人学习本质上难以扩展。我们认为，从野外人体视频中学习为机器人操控任务提供了一个有前景的解决方案，因为互联网上已经存在大量相关数据。在这项工作中，我们提出了VidBot，一个利用从野外单目RGB人体视频中学习到的三维可供性来实现零样本机器人操控的框架。VidBot采用一个流程从这些视频中提取显式表示，即视频中的三维手部轨迹，结合深度基础模型与运动恢复结构技术，重建出与具体具身形式无关、具有时间一致性和公制尺度的三维可供性表示。我们引入了一个从粗到细的可供性学习模型，该模型首先从像素空间识别粗粒度动作，然后通过一个扩散模型生成细粒度的交互轨迹，该扩散模型以粗粒度动作为条件，并受测试时约束引导以实现上下文感知的交互规划，从而能够对新场景和新具身形式实现显著的泛化。大量实验证明了VidBot的有效性，其在零样本设置下的13项操控任务中显著优于同类方法，并且可以无缝部署到现实环境中的不同机器人系统上。VidBot为利用日常人体视频使机器人学习更具可扩展性开辟了道路。