Large pre-trained models have had a significant impact on computer vision by enabling multi-modal learning, where the CLIP model has achieved impressive results in image classification, object detection, and semantic segmentation. However, the model's performance on 3D point cloud processing tasks is limited due to the domain gap between depth maps from 3D projection and training images of CLIP. This paper proposes DiffCLIP, a new pre-training framework that incorporates stable diffusion with ControlNet to minimize the domain gap in the visual branch. Additionally, a style-prompt generation module is introduced for few-shot tasks in the textual branch. Extensive experiments on the ModelNet10, ModelNet40, and ScanObjectNN datasets show that DiffCLIP has strong abilities for 3D understanding. By using stable diffusion and style-prompt generation, DiffCLIP achieves an accuracy of 43.2\% for zero-shot classification on OBJ\_BG of ScanObjectNN, which is state-of-the-art performance, and an accuracy of 80.6\% for zero-shot classification on ModelNet10, which is comparable to state-of-the-art performance.

翻译：大规模预训练模型通过支持多模态学习对计算机视觉产生了重要影响，其中CLIP模型在图像分类、目标检测和语义分割方面取得了令人瞩目的成果。然而，由于3D投影深度图与CLIP训练图像之间存在领域差距，该模型在3D点云处理任务上的性能受到限制。本文提出DiffCLIP，一种结合ControlNet稳定扩散的新预训练框架，旨在缩小视觉分支中的领域差距。此外，在文本分支中引入了一个风格提示生成模块以应对少样本任务。在ModelNet10、ModelNet40和ScanObjectNN数据集上的大量实验表明，DiffCLIP具备强大的3D理解能力。通过使用稳定扩散和风格提示生成，DiffCLIP在ScanObjectNN的OBJ_BG子集上实现了43.2%的零样本分类准确率，达到当前最优性能；在ModelNet10上实现了80.6%的零样本分类准确率，与当前最优性能相当。