Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.

翻译：传统三维场景理解方法依赖带标签的三维数据集训练单一任务的监督模型。我们提出OpenScene这一替代方法，通过模型预测三维场景点的密集特征，使其与CLIP特征空间中的文本和图像像素共同嵌入。这种零样本方法实现了任务无关训练和开放词汇查询。例如，为执行最先进的零样本三维语义分割，该方法首先推断每个三维点的CLIP特征，随后基于与任意类别标签嵌入的相似性进行分类。更有趣的是，它能实现一系列前所未有的开放词汇场景理解应用。例如，允许用户输入任意文本查询，随后生成热力图显示场景中匹配的区域。我们的方法能有效识别复杂三维场景中的物体、材质、功能属性、活动类型及房间类别，且所有功能均由单个模型在无任何三维标注数据训练下完成。