For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )

翻译：为使机器人能够执行多种任务，需要一种语义丰富且紧凑高效的三维世界表示，以支持任务驱动的感知与规划。近期研究尝试利用大型视觉语言模型的特征将语义编码到三维表示中，但这些方法通常生成逐点特征向量的地图，不仅难以在更大环境中扩展，也缺乏环境中实体间的语义空间关系——这种关系对下游规划十分有用。本文提出ConceptGraphs，一种面向三维场景的开放词汇图结构表示。该方法利用二维基础模型，通过多视图关联将其输出融合至三维空间。所得表示可泛化至未见语义类别，无需收集大规模三维数据集或微调模型。我们通过若干指定为抽象（语言）提示的下游规划任务展示了该表示的实用性，这些任务需要空间与语义概念的复杂推理。（项目页面：https://concept-graphs.github.io/ 讲解视频：https://youtu.be/mRhNkQwRYnc）