Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ, and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Project page is available at https://be2rlab.github.io/OSMa-Bench/.

翻译：开放语义建图（OSM）是机器人感知领域的一项关键技术，它结合了语义分割与SLAM技术。本文介绍了一种动态可配置、高度自动化的、基于LLM/LVLM的评估流程，用于评估OSM解决方案，该流程被称为OSMa-Bench（开放语义建图基准）。本研究重点评估了最先进的语义建图算法在变化的室内光照条件下的性能，这是室内环境中的一个关键挑战。我们引入了一个包含模拟RGB-D序列和真实3D重建的新数据集，便于对不同光照条件下的建图性能进行严格分析。通过对ConceptGraphs、BBQ和OpenScene等领先模型进行实验，我们评估了物体识别与分割的语义保真度。此外，我们引入了一种场景图评估方法，以分析模型解释语义结构的能力。实验结果揭示了这些模型的鲁棒性，为开发具有韧性和适应性的机器人系统指明了未来的研究方向。项目页面位于 https://be2rlab.github.io/OSMa-Bench/。