Vision-Language models like CLIP have been widely adopted for various tasks due to their impressive zero-shot capabilities. However, CLIP is not suitable for extracting 3D geometric features as it was trained on only images and text by natural language supervision. We work on addressing this limitation and propose a new framework termed CG3D (CLIP Goes 3D) where a 3D encoder is learned to exhibit zero-shot capabilities. CG3D is trained using triplets of pointclouds, corresponding rendered 2D images, and texts using natural language supervision. To align the features in a multimodal embedding space, we utilize contrastive loss on 3D features obtained from the 3D encoder, as well as visual and text features extracted from CLIP. We note that the natural images used to train CLIP and the rendered 2D images in CG3D have a distribution shift. Attempting to train the visual and text encoder to account for this shift results in catastrophic forgetting and a notable decrease in performance. To solve this, we employ prompt tuning and introduce trainable parameters in the input space to shift CLIP towards the 3D pre-training dataset utilized in CG3D. We extensively test our pre-trained CG3D framework and demonstrate its impressive capabilities in zero-shot, open scene understanding, and retrieval tasks. Further, it also serves as strong starting weights for fine-tuning in downstream 3D recognition tasks.

翻译：像CLIP这样的视觉-语言模型因其卓越的零样本能力而被广泛应用于各类任务。然而，由于CLIP仅通过自然语言监督在图像和文本上进行训练，它并不适用于提取三维几何特征。我们致力于解决这一局限，并提出一个新框架CG3D（CLIP迈向3D），其中学习了一个三维编码器以展现零样本能力。CG3D利用自然语言监督通过点云、对应渲染二维图像和文本的三元组进行训练。为了在多模态嵌入空间中对齐特征，我们对从三维编码器获得的三维特征以及从CLIP提取的视觉和文本特征使用对比损失。我们注意到，用于训练CLIP的自然图像与CG3D中渲染的二维图像存在分布偏移。试图通过训练视觉和文本编码器来补偿这种偏移会导致灾难性遗忘和性能显著下降。为解决此问题，我们采用提示调优，在输入空间引入可训练参数，使CLIP向CG3D中使用的三维预训练数据集偏移。我们广泛测试了预训练的CG3D框架，并展示了其在零样本、开放场景理解和检索任务中的卓越能力。此外，它还为下游三维识别任务的微调提供了强大的初始权重。