Recently, large-scale pre-trained models such as Segment-Anything Model (SAM) and Contrastive Language-Image Pre-training (CLIP) have demonstrated remarkable success and revolutionized the field of computer vision. These foundation vision models effectively capture knowledge from a large-scale broad data with their vast model parameters, enabling them to perform zero-shot segmentation on previously unseen data without additional training. While they showcase competence in 2D tasks, their potential for enhancing 3D scene understanding remains relatively unexplored. To this end, we present a novel framework that adapts various foundational models for the 3D point cloud segmentation task. Our approach involves making initial predictions of 2D semantic masks using different large vision models. We then project these mask predictions from various frames of RGB-D video sequences into 3D space. To generate robust 3D semantic pseudo labels, we introduce a semantic label fusion strategy that effectively combines all the results via voting. We examine diverse scenarios, like zero-shot learning and limited guidance from sparse 2D point labels, to assess the pros and cons of different vision foundation models. Our approach is experimented on ScanNet dataset for 3D indoor scenes, and the results demonstrate the effectiveness of adopting general 2D foundation models on solving 3D point cloud segmentation tasks.

翻译：近期，诸如Segment-Anything Model（SAM）和对比语言-图像预训练（CLIP）等大规模预训练模型取得了显著成功，并彻底改变了计算机视觉领域。这些基础视觉模型凭借其庞大的模型参数，有效从大规模广泛数据中捕获知识，从而能够对未见过的数据执行零样本分割，无需额外训练。尽管它们在2D任务中表现出色，但其在增强3D场景理解方面的潜力仍相对未被探索。为此，我们提出了一种新颖框架，将多种基础模型适配到3D点云分割任务中。我们的方法包括使用不同大型视觉模型对2D语义掩码进行初始预测，然后将这些来自RGB-D视频序列多帧的掩码预测投影到3D空间。为生成稳健的3D语义伪标签，我们引入了一种语义标签融合策略，通过投票有效整合所有结果。我们考察了零样本学习及稀疏2D点标签有限指导等多样场景，以评估不同视觉基础模型的优劣。我们的方法在ScanNet数据集上针对3D室内场景进行了实验，结果表明采用通用2D基础模型解决3D点云分割任务的有效性。