Recent work in visual representation learning for robotics demonstrates the viability of learning from large video datasets of humans performing everyday tasks. Leveraging methods such as masked autoencoding and contrastive learning, these representations exhibit strong transfer to policy learning for visuomotor control. But, robot learning encompasses a diverse set of problems beyond control including grasp affordance prediction, language-conditioned imitation learning, and intent scoring for human-robot collaboration, amongst others. First, we demonstrate that existing representations yield inconsistent results across these tasks: masked autoencoding approaches pick up on low-level spatial features at the cost of high-level semantics, while contrastive learning approaches capture the opposite. We then introduce Voltron, a framework for language-driven representation learning from human videos and associated captions. Voltron trades off language-conditioned visual reconstruction to learn low-level visual patterns, and visually-grounded language generation to encode high-level semantics. We also construct a new evaluation suite spanning five distinct robot learning problems $\unicode{x2013}$ a unified platform for holistically evaluating visual representations for robotics. Through comprehensive, controlled experiments across all five problems, we find that Voltron's language-driven representations outperform the prior state-of-the-art, especially on targeted problems requiring higher-level features.

翻译：摘要：近期针对机器人技术的视觉表征学习研究表明，通过大规模人类日常任务视频数据集进行学习具备可行性。利用掩码自编码与对比学习等方法，这些表征在视觉运动控制策略学习中展现出强大迁移能力。然而，机器人学习涵盖控制之外的多元问题，包括抓取可能性预测、语言条件模仿学习以及人机协作意图评分等。首先，我们发现现有表征在这些任务中呈现不一致性：掩码自编码方法虽捕捉低级空间特征，却牺牲了高级语义信息；而对比学习方法则呈现相反特性。基于此，我们提出Voltron——一种从人类视频及对应字幕中进行语言驱动表征学习的框架。Voltron通过权衡语言条件视觉重建以学习低级视觉模式，同时借助视觉基础语言生成编码高级语义。此外，我们构建了涵盖五个不同机器人学习问题的新评估套件——一个用于全面评估机器人视觉表征的统一平台。通过针对所有五个问题的综合性对照实验，我们发现Voltron的语言驱动表征优于先前的最优方法，尤其在需要高级特征的特定问题上表现突出。