Open-vocabulary segmentation of 3D scenes is a fundamental function of human perception and thus a crucial objective in computer vision research. However, this task is heavily impeded by the lack of large-scale and diverse 3D open-vocabulary segmentation datasets for training robust and generalizable models. Distilling knowledge from pre-trained 2D open-vocabulary segmentation models helps but it compromises the open-vocabulary feature as the 2D models are mostly finetuned with close-vocabulary datasets. We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. Specifically, given only the open-vocabulary text descriptions of the objects in a scene, we distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF), which effectively lifts 2D features into view-consistent 3D segmentation. A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process. Extensive experiments show that our method even outperforms fully supervised models trained with segmentation annotations in certain scenes, suggesting that 3D open-vocabulary segmentation can be effectively learned from 2D images and text-image pairs. Code is available at \url{https://github.com/Kunhao-Liu/3D-OVS}.

翻译：三维场景的开放词汇分割是人类感知的基本功能，因此也是计算机视觉研究的关键目标。然而，该任务因缺乏大规模多样化的三维开放词汇分割数据集来训练鲁棒且泛化的模型而严重受阻。从预训练的二维开放词汇分割模型中蒸馏知识虽能提供帮助，但会损害开放词汇特征，因为二维模型大多使用封闭词汇数据集进行微调。我们通过以弱监督方式利用预训练基础模型CLIP和DINO来应对三维开放词汇分割中的挑战。具体而言，仅凭场景中物体的开放词汇文本描述，我们将CLIP和DINO的开放词汇多模态知识与物体推理能力蒸馏到神经辐射场（NeRF）中，从而将二维特征有效地提升为视图一致的三维分割。本方法的一个显著特点是，无论是基础模型还是蒸馏过程均无需任何人工分割标注。大量实验表明，我们的方法甚至在某些场景中优于使用分割标注训练的全监督模型，这证明三维开放词汇分割可以从二维图像和文本-图像对中有效学习。代码已发布于\url{https://github.com/Kunhao-Liu/3D-OVS}。