For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few ($\sim$10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.

翻译：对于3D物体操作任务，构建显式3D表示的方法优于仅依赖相机图像的方法。但使用体素等显式3D表示会带来巨大的计算成本，影响其可扩展性。本文提出RVT——一种兼具可扩展性与准确性的多视角Transformer，用于3D操作任务。其关键特性包括跨视角信息聚合的注意力机制，以及基于机器人工作空间周围虚拟视角对相机输入进行重渲染。仿真实验表明，单一RVT模型可有效应对包含249种任务变体的18项RLBench任务，相对成功率比现有最优方法（PerAct）提升26%。达到同等性能时，训练速度比PerAct快36倍，推理速度达PerAct的2.3倍。此外，在真实世界中，RVT仅需每项任务约10次示范即可执行多种操作任务。可视化结果、代码及预训练模型已发布于https://robotic-view-transformer.github.io/。