The field of visual representation learning has seen explosive growth in the past years, but its benefits in robotics have been surprisingly limited so far. Prior work uses generic visual representations as a basis to learn (task-specific) robot action policies (e.g. via behavior cloning). While the visual representations do accelerate learning, they are primarily used to encode visual observations. Thus, action information has to be derived purely from robot data, which is expensive to collect! In this work, we present a scalable alternative where the visual representations can help directly infer robot actions. We observe that vision encoders express relationships between image observations as distances (e.g. via embedding dot product) that could be used to efficiently plan robot behavior. We operationalize this insight and develop a simple algorithm for acquiring a distance function and dynamics predictor, by fine-tuning a pre-trained representation on human collected video sequences. The final method is able to substantially outperform traditional robot learning baselines (e.g. 70% success v.s. 50% for behavior cloning on pick-place) on a suite of diverse real-world manipulation tasks. It can also generalize to novel objects, without using any robot demonstrations during train time. For visualizations of the learned policies please check: https://agi-labs.github.io/manipulate-by-seeing/

翻译：视觉表征学习领域在过去几年经历了爆发式发展，但其在机器人领域的应用效益却出奇有限。现有方法将通用视觉表征作为学习（任务特定）机器人动作策略的基础（例如通过行为克隆）。虽然视觉表征确实加速了学习过程，但它们主要被用于编码视觉观测。因此，动作信息必须完全从昂贵的机器人数据中推导得出！本文提出一种可扩展的替代方案：视觉表征可直接辅助推断机器人动作。我们观察到视觉编码器能将图像观测之间的关系表达为距离形式（例如通过嵌入点积），这可用于高效规划机器人行为。我们将这一见解付诸实践，开发出一种简单的算法：通过对人类采集视频序列上的预训练表征进行微调，获取距离函数与动力学预测器。最终方法在一系列多样化的真实世界操作任务中显著优于传统机器人学习基线（例如拾放任务中成功率70%对比行为克隆的50%），且无需在训练阶段使用任何机器人示教即可泛化至新物体。学习策略的可视化演示请参见：https://agi-labs.github.io/manipulate-by-seeing/