Visual robotic manipulation research and applications often use multiple cameras, or views, to better perceive the world. How else can we utilize the richness of multi-view data? In this paper, we investigate how to learn good representations with multi-view data and utilize them for visual robotic manipulation. Specifically, we train a multi-view masked autoencoder which reconstructs pixels of randomly masked viewpoints and then learn a world model operating on the representations from the autoencoder. We demonstrate the effectiveness of our method in a range of scenarios, including multi-view control and single-view control with auxiliary cameras for representation learning. We also show that the multi-view masked autoencoder trained with multiple randomized viewpoints enables training a policy with strong viewpoint randomization and transferring the policy to solve real-robot tasks without camera calibration and an adaptation procedure. Video demonstrations are available at: https://sites.google.com/view/mv-mwm.

翻译：视觉机器人操控的研究与应用中，常使用多个摄像头（即多视角）来更好地感知世界。我们还能如何利用多视角数据的丰富性？本文研究了如何利用多视角数据学习有效表示，并将其应用于视觉机器人操控。具体而言，我们训练了一个多视角掩码自编码器，用于重建随机掩码视角的像素，并在此基础上学习一个基于自编码器表示的世界模型。我们在多种场景中验证了该方法的效果，包括多视角控制以及利用辅助摄像头进行表示学习的单视角控制。我们还表明，通过使用多个随机化视角训练的多视角掩码自编码器，能够训练出具备强视角随机化能力的策略，并在无需相机标定和自适应过程的情况下，将该策略迁移至真实机器人任务中。视频演示请参见：https://sites.google.com/view/mv-mwm。