Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN.

翻译：掩码自编码器（MAE）在2D与3D计算机视觉的自监督学习中展现出优异性能。然而，现有MAE类方法仅能学习单一模态数据（即图像或点云），忽略了2D与3D之间隐含的语义与几何关联。本文探索了2D模态如何促进3D掩码自编码，并提出Joint-MAE——一种面向自监督3D点云预训练的2D-3D联合MAE框架。Joint-MAE对输入3D点云及其投影的2D图像进行随机掩码，随后重构两模态的掩码信息。为实现更好的跨模态交互，我们通过两个层次化2D-3D嵌入模块、一个联合编码器及一个包含模态共享与模态特定解码器的联合解码器构建Joint-MAE。在此基础上，我们进一步引入两种跨模态策略以增强3D表征学习：用于2D-3D语义线索的局部对齐注意力机制，以及用于2D-3D几何约束的交叉重构损失。通过我们的预训练范式，Joint-MAE在多项下游任务上取得了优越性能，例如在ModelNet40上线性SVM准确率达92.4%，在ScanObjectNN最困难子集上准确率达86.07%。