Large pre-trained models have had a significant impact on computer vision by enabling multi-modal learning, where the CLIP model has achieved impressive results in image classification, object detection, and semantic segmentation. However, the model's performance on 3D point cloud processing tasks is limited due to the domain gap between depth maps from 3D projection and training images of CLIP. This paper proposes DiffCLIP, a new pre-training framework that incorporates stable diffusion with ControlNet to minimize the domain gap in the visual branch. Additionally, a style-prompt generation module is introduced for few-shot tasks in the textual branch. Extensive experiments on the ModelNet10, ModelNet40, and ScanObjectNN datasets show that DiffCLIP has strong abilities for 3D understanding. By using stable diffusion and style-prompt generation, DiffCLIP achieves an accuracy of 43.2\% for zero-shot classification on OBJ\_BG of ScanObjectNN, which is state-of-the-art performance, and an accuracy of 80.6\% for zero-shot classification on ModelNet10, which is comparable to state-of-the-art performance.

翻译：大型预训练模型通过支持多模态学习，对计算机视觉产生了重大影响，其中CLIP模型在图像分类、目标检测和语义分割中取得了显著成果。然而，由于三维投影深度图与CLIP训练图像之间存在域差距，该模型在三维点云处理任务上的性能受到限制。本文提出DiffCLIP，一种结合ControlNet的稳定扩散模型的新型预训练框架，以最小化视觉分支中的域差距。此外，在文本分支中引入风格提示生成模块，用于少样本任务。在ModelNet10、ModelNet40和ScanObjectNN数据集上的大量实验表明，DiffCLIP具备强大的三维理解能力。通过使用稳定扩散和风格提示生成，DiffCLIP在ScanObjectNN的OBJ_BG子集上实现43.2%的零样本分类准确率，达到最优性能；在ModelNet10上实现80.6%的零样本分类准确率，与最新最优性能相当。