Humans excel at forming mental maps of their surroundings, equipping them to understand object relationships and navigate based on language queries. Our previous work, SI Maps (Nanwani L, Agarwal A, Jain K, et al. Instance-level semantic maps for vision language navigation. In: 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE; 2023 Aug.), showed that having instance-level information and the semantic understanding of an environment helps significantly improve performance for language-guided tasks. We extend this instance-level approach to 3D while increasing the pipeline's robustness and improving quantitative and qualitative results. Our method leverages foundational models for object recognition, image segmentation, and feature extraction. We propose a representation that results in a 3D point cloud map with instance-level embeddings, which bring in the semantic understanding that natural language commands can query. Quantitatively, the work improves upon the success rate of language-guided tasks. At the same time, we qualitatively observe the ability to identify instances more clearly and leverage the foundational models and language and image-aligned embeddings to identify objects that, otherwise, a closed-set approach wouldn't be able to identify. Project Page - https://smart-wheelchair-rrc.github.io/o3d-sim-webpage

翻译：人类擅长构建周围环境的心理地图，使其能够理解物体间的关系并根据语言指令进行导航。我们先前的工作SI Maps（Nanwani L, Agarwal A, Jain K等. 面向视觉语言导航的实例级语义地图. 见：2023年第32届IEEE机器人与人类交互通信国际会议（RO-MAN）. IEEE; 2023年8月）表明，获取环境的实例级信息与语义理解能显著提升语言引导任务的性能。我们将此实例级方法扩展至三维空间，同时增强了流程的鲁棒性并改善了定量与定性结果。我们的方法利用基础模型进行物体识别、图像分割与特征提取。我们提出了一种表征方法，可生成带有实例级嵌入的三维点云地图，这种嵌入引入了自然语言指令可查询的语义理解。定量实验表明，该工作提升了语言引导任务的成功率。同时，我们通过定性观察发现，该方法能更清晰地识别实例，并利用基础模型以及语言与图像对齐的嵌入来识别物体——这些物体在封闭集方法中无法被识别。项目页面 - https://smart-wheelchair-rrc.github.io/o3d-sim-webpage