Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for both 2D and 3D computer vision. However, existing MAE-style methods can only learn from the data of a single modality, i.e., either images or point clouds, which neglect the implicit semantic and geometric correlation between 2D and 3D. In this paper, we explore how the 2D modality can benefit 3D masked autoencoding, and propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training. Joint-MAE randomly masks an input 3D point cloud and its projected 2D images, and then reconstructs the masked information of the two modalities. For better cross-modal interaction, we construct our JointMAE by two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and model-specific decoders. On top of this, we further introduce two cross-modal strategies to boost the 3D representation learning, which are local-aligned attention mechanisms for 2D-3D semantic cues, and a cross-reconstruction loss for 2D-3D geometric constraints. By our pre-training paradigm, Joint-MAE achieves superior performance on multiple downstream tasks, e.g., 92.4% accuracy for linear SVM on ModelNet40 and 86.07% accuracy on the hardest split of ScanObjectNN.

翻译：掩码自编码器（MAE）在二维和三维计算机视觉的自监督学习中均展现了优异性能。然而，现有基于MAE的方法仅能学习单一模态数据（即图像或点云），忽略了二维与三维之间隐含的语义与几何关联。本文探索了二维模态如何促进三维掩码自编码任务，并提出Joint-MAE——一种用于自监督三维点云预训练的2D-3D联合MAE框架。Joint-MAE随机掩码输入的三维点云及其投影的二维图像，随后重建两种模态的掩码信息。为实现更好的跨模态交互，我们通过两个分层2D-3D嵌入模块、一个联合编码器以及包含模态共享与模态特异性解码器的联合解码器构建了Joint-MAE。在此基础上，我们进一步引入两种跨模态策略以增强三维表示学习：用于提取2D-3D语义线索的局部对齐注意力机制，以及施加2D-3D几何约束的交叉重建损失。通过本预训练范式，Joint-MAE在多项下游任务中取得了优越性能，例如在ModelNet40上线性SVM准确率达92.4%，在ScanObjectNN最难子集上准确率达86.07%。