Open Semantic Mapping (OSM) is a key technology in robotic perception, combining semantic segmentation and SLAM techniques. This paper introduces a dynamically configurable and highly automated LLM/LVLM-powered pipeline for evaluating OSM solutions called OSMa-Bench (Open Semantic Mapping Benchmark). The study focuses on evaluating state-of-the-art semantic mapping algorithms under varying indoor lighting conditions, a critical challenge in indoor environments. We introduce a novel dataset with simulated RGB-D sequences and ground truth 3D reconstructions, facilitating the rigorous analysis of mapping performance across different lighting conditions. Through experiments on leading models such as ConceptGraphs, BBQ and OpenScene, we evaluate the semantic fidelity of object recognition and segmentation. Additionally, we introduce a Scene Graph evaluation method to analyze the ability of models to interpret semantic structure. The results provide insights into the robustness of these models, forming future research directions for developing resilient and adaptable robotic systems. Our code is available at https://be2rlab.github.io/OSMa-Bench/.

翻译：开放语义建图（OSM）是机器人感知中的一项关键技术，它结合了语义分割与SLAM技术。本文介绍了一种动态可配置、高度自动化的LLM/LVLM驱动流水线，用于评估OSM解决方案，该流水线称为OSMa-Bench（开放语义建图基准）。本研究重点评估了在变化的室内光照条件下最先进的语义建图算法，这是室内环境中的一个关键挑战。我们引入了一个包含模拟RGB-D序列和真实三维重建结果的新数据集，便于对不同光照条件下的建图性能进行严格分析。通过对ConceptGraphs、BBQ和OpenScene等领先模型进行实验，我们评估了物体识别与分割的语义保真度。此外，我们引入了一种场景图评估方法，以分析模型解析语义结构的能力。实验结果揭示了这些模型的鲁棒性，为开发具有韧性和适应性的机器人系统指明了未来的研究方向。我们的代码可在 https://be2rlab.github.io/OSMa-Bench/ 获取。