While large-scale robotic systems typically rely on textual instructions for tasks, this work explores a different approach: can robots infer the task directly from observing humans? This shift necessitates the robot's ability to decode human intent and translate it into executable actions within its physical constraints and environment. We introduce Vid2Robot, a novel end-to-end video-based learning framework for robots. Given a video demonstration of a manipulation task and current visual observations, Vid2Robot directly produces robot actions. This is achieved through a unified representation model trained on a large dataset of human video and robot trajectory. The model leverages cross-attention mechanisms to fuse prompt video features to the robot's current state and generate appropriate actions that mimic the observed task. To further improve policy performance, we propose auxiliary contrastive losses that enhance the alignment between human and robot video representations. We evaluate Vid2Robot on real-world robots, demonstrating a 20% improvement in performance compared to other video-conditioned policies when using human demonstration videos. Additionally, our model exhibits emergent capabilities, such as successfully transferring observed motions from one object to another, and long-horizon composition, thus showcasing its potential for real-world applications. Project website: vid2robot.github.io

翻译：虽然大规模机器人系统通常依赖文本指令执行任务，但本研究探索了另一种范式：机器人能否通过观察人类行为直接推断任务目标？这种转变要求机器人具备解码人类意图的能力，并能将其转化为物理约束与环境条件下的可执行动作。我们提出Vid2Robot——一种新颖的端到端视频驱动机器人学习框架。给定操作任务的视频演示与当前视觉观测，Vid2Robot可直接生成机器人动作序列。该框架通过在大规模人类视频与机器人轨迹数据集上训练的统表征模型实现，利用交叉注意力机制将提示视频特征与机器人当前状态相融合，生成模仿观测任务的合理动作。为进一步提升策略性能，我们提出辅助对比损失函数以增强人类与机器人视频表征的对齐效果。在真实机器人系统上的评估表明，相较其他视频条件策略，Vid2Robot在使用人类演示视频时性能提升20%。此外，模型展现出涌现能力：成功实现观测动作在物体间的迁移以及长时序任务组合，展示了其在真实世界应用中的潜力。项目网站：vid2robot.github.io