Recently, groundbreaking results have been presented on open-vocabulary semantic image segmentation. Such methods segment each pixel in an image into arbitrary categories provided at run-time in the form of text prompts, as opposed to a fixed set of classes defined at training time. In this work, we present a zero-shot volumetric open-vocabulary semantic scene segmentation method. Our method builds on the insight that we can fuse image features from a vision-language model into a neural implicit representation. We show that the resulting feature field can be segmented into different classes by assigning points to natural language text prompts. The implicit volumetric representation enables us to segment the scene both in 3D and 2D by rendering feature maps from any given viewpoint of the scene. We show that our method works on noisy real-world data and can run in real-time on live sensor data dynamically adjusting to text prompts. We also present quantitative comparisons on the ScanNet dataset.

翻译：近期，开放词汇语义图像分割领域取得了突破性成果。此类方法能在运行时以文本提示形式，将图像中的每个像素分割为任意类别，而非局限于训练时定义的固定类别集合。本文提出一种零样本体积开放词汇语义场景分割方法。该方法基于一个关键洞察：可将视觉-语言模型的图像特征融合到神经隐式表示中。我们证明，通过将点云与自然语言文本提示关联，所得特征场可被分割为不同类别。隐式体积表示支持通过从场景任意视角渲染特征图，实现3D与2D场景分割。实验表明，本方法可处理含噪真实数据，并能在实时传感器数据上动态响应文本提示。我们还在ScanNet数据集上给出了定量对比结果。