Learning from demonstration is a powerful method for teaching robots new skills, and more demonstration data often improves policy learning. However, the high cost of collecting demonstration data is a significant bottleneck. Videos, as a rich data source, contain knowledge of behaviors, physics, and semantics, but extracting control-specific information from them is challenging due to the lack of action labels. In this work, we introduce a novel framework, Any-point Trajectory Modeling (ATM), that utilizes video demonstrations by pre-training a trajectory model to predict future trajectories of arbitrary points within a video frame. Once trained, these trajectories provide detailed control guidance, enabling the learning of robust visuomotor policies with minimal action-labeled data. Our method's effectiveness is demonstrated across 130 simulation tasks, focusing on language-conditioned manipulation tasks. Visualizations and code are available at: \url{https://xingyu-lin.github.io/atm}.

翻译：从示范中学习是一种教授机器人新技能的有效方法，且更多的示范数据通常能提升策略学习效果。然而，收集示范数据的高昂成本是一个重大瓶颈。视频作为一种丰富的数据源，蕴含行为、物理和语义知识，但由于缺乏动作标签，从中提取控制特定信息具有挑战性。在这项工作中，我们提出了一种新颖框架——任意点轨迹建模（ATM），该框架通过预训练一个轨迹模型来预测视频帧内任意点的未来轨迹，从而利用视频示范。训练完成后，这些轨迹提供详细的控制指导，使得仅需极少的动作标注数据就能学习稳健的视觉运动策略。我们方法的效果在130个模拟任务中得到了验证，重点关注语言条件操控任务。相关可视化结果和代码可访问：\url{https://xingyu-lin.github.io/atm}。