Generalizable 3D part segmentation is important but challenging in vision and robotics. Training deep models via conventional supervised methods requires large-scale 3D datasets with fine-grained part annotations, which are costly to collect. This paper explores an alternative way for low-shot part segmentation of 3D point clouds by leveraging a pretrained image-language model, GLIP, which achieves superior performance on open-vocabulary 2D detection. We transfer the rich knowledge from 2D to 3D through GLIP-based part detection on point cloud rendering and a novel 2D-to-3D label lifting algorithm. We also utilize multi-view 3D priors and few-shot prompt tuning to boost performance significantly. Extensive evaluation on PartNet and PartNet-Mobility datasets shows that our method enables excellent zero-shot 3D part segmentation. Our few-shot version not only outperforms existing few-shot approaches by a large margin but also achieves highly competitive results compared to the fully supervised counterpart. Furthermore, we demonstrate that our method can be directly applied to iPhone-scanned point clouds without significant domain gaps.

翻译：通用化的三维部件分割在视觉与机器人领域中具有重要意义但极具挑战性。通过传统监督方法训练深度模型需要大规模且带有精细部件标注的三维数据集，而此类数据集的构建成本高昂。本文探索了一种利用预训练图像-语言模型GLIP（该模型在开放词汇二维检测中表现优异）实现三维点云低样本部件分割的替代方案。我们通过基于GLIP的点云渲染部件检测及一种新型二维到三维标签提升算法，将丰富的二维知识迁移至三维领域。同时，我们利用多视角三维先验和少样本提示调优显著提升性能。在PartNet与PartNet-Mobility数据集上的广泛评估表明，该方法可实现卓越的零样本三维部件分割。我们提出的少样本版本不仅以显著优势超越现有少样本方法，且与全监督方法相比也展现出极具竞争力的结果。此外，我们还证明该方法可直接应用于iPhone扫描的点云，而不会产生显著的域差异。