With the overwhelming trend of mask image modeling led by MAE, generative pre-training has shown a remarkable potential to boost the performance of fundamental models in 2D vision. However, in 3D vision, the over-reliance on Transformer-based backbones and the unordered nature of point clouds have restricted the further development of generative pre-training. In this paper, we propose a novel 3D-to-2D generative pre-training method that is adaptable to any point cloud model. We propose to generate view images from different instructed poses via the cross-attention mechanism as the pre-training scheme. Generating view images has more precise supervision than its point cloud counterpart, thus assisting 3D backbones to have a finer comprehension of the geometrical structure and stereoscopic relations of the point cloud. Experimental results have proved the superiority of our proposed 3D-to-2D generative pre-training over previous pre-training methods. Our method is also effective in boosting the performance of architecture-oriented approaches, achieving state-of-the-art performance when fine-tuning on ScanObjectNN classification and ShapeNetPart segmentation tasks. Code is available at https://github.com/wangzy22/TAP.

翻译：在MAE主导的掩码图像建模趋势下，生成式预训练已展现出提升二维视觉基础模型性能的巨大潜力。然而在三维视觉领域，对基于Transformer主干网络的过度依赖以及点云的无序性，限制了生成式预训练的进一步发展。本文提出一种新颖的3D到2D生成式预训练方法，该方法可适配任意点云模型。我们利用交叉注意力机制，通过生成不同指定姿态下的视角图像作为预训练方案。相较于点云形式的预训练，生成视角图像能提供更精确的监督信号，从而帮助三维主干网络更精细地理解点云的几何结构与立体关系。实验结果证明，我们提出的3D到2D生成式预训练方法优于以往的预训练方法。该方法还能有效提升基于架构的方法的性能，在ScanObjectNN分类任务和ShapeNetPart分割任务的微调中达到当前最优水平。代码已开源至https://github.com/wangzy22/TAP。