Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D zero-shot and few-shot learning. Despite the impressive performance in 2D, applying CLIP to help the learning in 3D scene understanding has yet to be explored. In this paper, we make the first attempt to investigate how CLIP knowledge benefits 3D scene understanding. We propose CLIP2Scene, a simple yet effective framework that transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network. We show that the pre-trained 3D network yields impressive performance on various downstream tasks, i.e., annotation-free and fine-tuning with labelled data for semantic segmentation. Specifically, built upon CLIP, we design a Semantic-driven Cross-modal Contrastive Learning framework that pre-trains a 3D network via semantic and spatial-temporal consistency regularization. For the former, we first leverage CLIP's text semantics to select the positive and negative point samples and then employ the contrastive loss to train the 3D network. In terms of the latter, we force the consistency between the temporally coherent point cloud features and their corresponding image features. We conduct experiments on SemanticKITTI, nuScenes, and ScanNet. For the first time, our pre-trained network achieves annotation-free 3D semantic segmentation with 20.8% and 25.08% mIoU on nuScenes and ScanNet, respectively. When fine-tuned with 1% or 100% labelled data, our method significantly outperforms other self-supervised methods, with improvements of 8% and 1% mIoU, respectively. Furthermore, we demonstrate the generalizability for handling cross-domain datasets. Code is publicly available https://github.com/runnanchen/CLIP2Scene.

翻译：对比语言-图像预训练（CLIP）在二维零样本和少样本学习中取得了令人瞩目的成果。尽管CLIP在二维任务上表现优异，但将其应用于辅助3D场景理解的学习尚未得到探索。本文首次尝试研究CLIP知识如何助力3D场景理解。我们提出CLIP2Scene，一个简单而有效的框架，将CLIP知识从二维图像-文本预训练模型迁移至三维点云网络。研究表明，预训练的三维网络在多种下游任务（即基于语义分割的无标注学习和标注数据微调）中展现出卓越性能。具体而言，基于CLIP，我们设计了一种语义驱动的跨模态对比学习框架，通过语义一致性和时空一致性正则化对三维网络进行预训练。对于前者，我们首先利用CLIP的文本语义选择正负点样本，然后采用对比损失训练三维网络。对于后者，我们强制时间上连续的点云特征与其对应图像特征之间的一致性。我们在SemanticKITTI、nuScenes和ScanNet数据集上进行了实验。首次实验表明，我们的预训练网络在nuScenes和ScanNet上实现了无标注的3D语义分割，mIoU分别达到20.8%和25.08%。当使用1%或100%的标注数据进行微调时，我们的方法显著优于其他自监督方法，mIoU分别提升了8%和1%。此外，我们证明了该方法在处理跨域数据集时的泛化能力。代码已公开于https://github.com/runnanchen/CLIP2Scene。