Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs

翻译：构建环境的三维地图是机器人导航、规划以及与场景中物体交互的核心。现有大多数将语义概念与三维地图集成的方法仍局限于封闭集设定：它们只能推理训练时预定义的有限概念集。此外，这些地图仅能通过类别标签进行查询，或近期的工作中通过文本提示进行查询。我们通过ConceptFusion解决了这两个问题，该场景表示（1）本质上是开放集的，能够推理超出封闭概念集的内容；（2）固有地支持多模态，使得三维地图能够支持从语言、图像、音频到三维几何的多样化查询，且所有查询方式协同工作。ConceptFusion利用当前在互联网规模数据上预训练的基石模型的开放集能力，推理跨模态（如自然语言、图像和音频）的概念。我们证明，像素对齐的开放集特征可以通过传统SLAM和多视图融合方法融合到三维地图中。这使得有效的零样本空间推理成为可能，无需任何额外训练或微调，并且比监督方法更好地保留了长尾概念，在三维交并比上性能提升超过40%。我们在多个真实世界数据集、模拟家庭环境、真实桌面操作任务以及自动驾驶平台上对ConceptFusion进行了广泛评估。我们展示了将基石模型与三维开放集多模态建图融合的新途径。更多信息请访问项目页面https://concept-fusion.github.io或观看5分钟讲解视频https://www.youtube.com/watch?v=rkXgws8fiDs。