The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models are released at https://github.com/salesforce/ULIP.

翻译：当前最先进三维模型的识别能力受限于标注数据量小且类别预设的数据集。在二维领域，最新进展表明此类问题可通过利用语言等其他模态的知识得到显著缓解。受此启发，在受限数据环境下借助多模态信息提升三维理解具有广阔前景，但这一研究方向尚未得到充分探索。为此，我们提出ULIP，通过预训练从图像、文本和三维点云三种模态中提取的对象三元组，学习三者的统一表征。为克服训练三元组短缺问题，ULIP利用已通过海量图像-文本对训练、具备通用视觉-文本空间的预训练视觉语言模型，并通过少量自动合成的三元组，学习与通用图像-文本空间对齐的三维表征空间。ULIP与三维骨干网络无关，可轻松集成至任意三维架构。实验表明，仅需使用ShapeNet55数据集通过ULIP框架预训练，即可有效提升多种最新三维骨干网络的性能，在ModelNet40和ScanObjectNN的标准三维分类与零样本三维分类任务上均达到最先进水平。在ScanObjectNN的三维分类任务中，ULIP使PointMLP性能提升约3%；在ModelNet40的零样本三维分类中，其top-1准确率较PointCLIP提升28.8%。我们的代码与预训练模型已开源至https://github.com/salesforce/ULIP。