Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature significantly as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting the open-vocabulary multimodal knowledge and object reasoning capability of pre-trained foundation models CLIP and DINO, without necessitating any fine-tuning. Specifically, we distill open-vocabulary visual and textual knowledge from CLIP into a neural radiance field (NeRF) which effectively lifts 2D features into view-consistent 3D segmentation. Furthermore, we introduce the Relevancy-Distribution Alignment loss and Feature-Distribution Alignment loss to respectively mitigate the ambiguities of CLIP features and distill precise object boundaries from DINO features, eliminating the need for segmentation annotations during training. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text-image pairs.

翻译：开放词汇的3D场景分割是人类感知的基本功能，也是计算机视觉研究的重要目标。然而，这一任务因缺乏大规模、多样化的3D开放词汇分割数据集来训练鲁棒且泛化的模型而严重受阻。从预训练的2D开放词汇分割模型中蒸馏知识虽有所助益，但因2D模型大多基于封闭词汇数据集微调，会显著损害开放词汇特征。我们通过利用预训练基础模型CLIP和DINO的开放词汇多模态知识与物体推理能力，在无需任何微调的情况下应对3D开放词汇分割的挑战。具体而言，我们将CLIP的开放词汇视觉与文本知识蒸馏至神经辐射场（NeRF），从而将2D特征有效提升为视角一致的3D分割。此外，我们引入相关性分布对齐损失和特征分布对齐损失，分别缓解CLIP特征的模糊性并从DINO特征中提取精确的物体边界，从而消除了训练过程中对分割标注的需求。大量实验表明，我们的方法甚至优于使用分割标注训练的完全监督模型，这证明了3D开放词汇分割可以从2D图像和文本-图像对中有效学习。