We present a large empirical investigation on the use of pre-trained visual representations (PVRs) for training downstream policies that execute real-world tasks. Our study spans five different PVRs, two different policy-learning paradigms (imitation and reinforcement learning), and three different robots for 5 distinct manipulation and indoor navigation tasks. From this effort, we can arrive at three insights: 1) the performance trends of PVRs in the simulation are generally indicative of their trends in the real world, 2) the use of PVRs enables a first-of-its-kind result with indoor ImageNav (zero-shot transfer to a held-out scene in the real world), and 3) the benefits from variations in PVRs, primarily data-augmentation and fine-tuning, also transfer to the real-world performance. See project website for additional details and visuals.
翻译:我们通过大规模实证研究,系统探究了预训练视觉表征(PVRs)在训练执行真实任务的下游策略中的应用。本研究涵盖五种不同的PVR、两种策略学习范式(模仿学习与强化学习),以及三台机器人执行的五项独立操作与室内导航任务。基于此研究,我们得出三点核心发现:1)PVR在模拟环境中的性能趋势通常能反映其在真实世界中的表现趋势;2)利用PVR首次实现了室内图像导航领域的突破性成果(在真实世界中对未见过场景的零样本迁移);3)PVR在不同变体(主要是数据增强与微调)中获得的性能提升亦可迁移至真实场景。更多细节与可视化内容请参见项目官网。