As two fundamental representation modalities of 3D objects, 3D point clouds and multi-view 2D images record shape information from different domains of geometric structures and visual appearances. In the current deep learning era, remarkable progress in processing such two data modalities has been achieved through respectively customizing compatible 3D and 2D network architectures. However, unlike multi-view image-based 2D visual modeling paradigms, which have shown leading performance in several common 3D shape recognition benchmarks, point cloud-based 3D geometric modeling paradigms are still highly limited by insufficient learning capacity, due to the difficulty of extracting discriminative features from irregular geometric signals. In this paper, we explore the possibility of boosting deep 3D point cloud encoders by transferring visual knowledge extracted from deep 2D image encoders under a standard teacher-student distillation workflow. Generally, we propose PointMCD, a unified multi-view cross-modal distillation architecture, including a pretrained deep image encoder as the teacher and a deep point encoder as the student. To perform heterogeneous feature alignment between 2D visual and 3D geometric domains, we further investigate visibility-aware feature projection (VAFP), by which point-wise embeddings are reasonably aggregated into view-specific geometric descriptors. By pair-wisely aligning multi-view visual and geometric descriptors, we can obtain more powerful deep point encoders without exhausting and complicated network modification. Experiments on 3D shape classification, part segmentation, and unsupervised learning strongly validate the effectiveness of our method. The code and data will be publicly available at https://github.com/keeganhk/PointMCD.

翻译：作为三维物体的两种基本表示模态，三维点云和多视图二维图像分别从几何结构与视觉外观的不同领域记录形状信息。在当前深度学习时代，通过分别为三维和二维数据定制兼容的网络架构，处理这两种数据模态已取得显著进展。然而，与多视图图像驱动的二维视觉建模范式（已在多个常见三维形状识别基准中展现出领先性能）不同，基于点云的三维几何建模范式仍受限于学习能力不足，这源于从不规则几何信号中提取判别性特征的困难。本文探索了在标准师生蒸馏框架下，通过迁移从深度二维图像编码器中提取的视觉知识来增强深度三维点云编码器的可能性。我们提出了PointMCD——一种统一的多视图跨模态蒸馏架构，包含一个预训练深度图像编码器作为教师网络和一个深度点编码器作为学生网络。为实现二维视觉域与三维几何域间的异构特征对齐，我们进一步研究了可见性感知特征投影（VAFP），通过该机制将逐点嵌入合理聚合为视图特定的几何描述符。通过成对对齐多视图视觉与几何描述符，我们能够在无需繁琐复杂的网络修改下获得更强大的深度点云编码器。在三维形状分类、部件分割及无监督学习上的实验充分验证了方法的有效性。代码与数据将在https://github.com/keeganhk/PointMCD 公开。