Recent works have shown that visual pretraining on egocentric datasets using masked autoencoders (MAE) can improve generalization for downstream robotics tasks. However, these approaches pretrain only on 2D images, while many robotics applications require 3D scene understanding. In this work, we propose 3D-MVP, a novel approach for 3D multi-view pretraining using masked autoencoders. We leverage Robotic View Transformer (RVT), which uses a multi-view transformer to understand the 3D scene and predict gripper pose actions. We split RVT's multi-view transformer into visual encoder and action decoder, and pretrain its visual encoder using masked autoencoding on large-scale 3D datasets such as Objaverse. We evaluate 3D-MVP on a suite of virtual robot manipulation tasks and demonstrate improved performance over baselines. We also show promising results on a real robot platform with minimal finetuning. Our results suggest that 3D-aware pretraining is a promising approach to improve sample efficiency and generalization of vision-based robotic manipulation policies. We will release code and pretrained models for 3D-MVP to facilitate future research. Project site: https://jasonqsy.github.io/3DMVP

翻译：近期研究表明，利用掩码自编码器（MAE）在以自我为中心的数据集上进行视觉预训练，能够提升下游机器人任务的泛化能力。然而，这些方法仅在二维图像上进行预训练，而许多机器人应用需要三维场景理解。本工作提出3D-MVP，一种基于掩码自编码器的三维多视图预训练新方法。我们利用机器人视图变换器（RVT）——该模型通过多视图变换器理解三维场景并预测夹爪位姿动作——将其多视图变换器拆分为视觉编码器与动作解码器，并在大规模三维数据集（如Objaverse）上通过掩码自编码对视觉编码器进行预训练。我们在系列虚拟机器人操作任务上评估3D-MVP，其性能表现优于基线方法。同时，在仅需少量微调的真实机器人平台上亦展现出良好效果。实验结果表明，三维感知预训练是提升基于视觉的机器人操作策略样本效率与泛化能力的有效途径。我们将开源3D-MVP的代码与预训练模型以促进后续研究。项目主页：https://jasonqsy.github.io/3DMVP