Visual-language models (VLMs) have recently been introduced in robotic mapping by using the latent representations, i.e., embeddings, of the VLMs to represent the natural language semantics in the map. The main benefit is moving beyond a small set of human-created labels toward open-vocabulary scene understanding. While there is anecdotal evidence that maps built this way support downstream tasks, such as navigation, rigorous analysis of the quality of the maps using these embeddings is lacking. We investigate two critical properties of map quality: queryability and consistency. The evaluation of queryability addresses the ability to retrieve information from the embeddings. We investigate two aspects of consistency: intra-map consistency and inter-map consistency. Intra-map consistency captures the ability of the embeddings to represent abstract semantic classes, and inter-map consistency captures the generalization properties of the representation. In this paper, we propose a way to analyze the quality of maps created using VLMs, which forms an open-source benchmark to be used when proposing new open-vocabulary map representations. We demonstrate the benchmark by evaluating the maps created by two state-of-the-art methods, VLMaps and OpenScene, using two encoders, LSeg and OpenSeg, using real-world data from the Matterport3D data set. We find that OpenScene outperforms VLMaps with both encoders, and LSeg outperforms OpenSeg with both methods.

翻译：视觉-语言模型（VLM）最近被引入机器人建图领域，通过使用VLM的潜在表示（即嵌入向量）来表征地图中的自然语言语义。其主要优势在于突破有限的人工标注标签，实现开放词汇场景理解。尽管有经验性证据表明，此类地图能支持导航等下游任务，但对基于嵌入向量的地图质量缺乏严谨分析。我们研究了地图质量的两个关键特性：可查询性与一致性。可查询性评估衡量从嵌入中检索信息的能力，一致性评估则包含地图内一致性与地图间一致性：前者指嵌入向量表征抽象语义类别的能力，后者指表征的泛化性能。本文提出了一种分析VLM建图质量的方法，并构建了开源基准测试，可用于验证新型开放词汇地图表征方案。我们通过Matterport3D数据集中的真实世界数据，利用LSeg和OpenSeg两种编码器，对VLMaps与OpenScene两种前沿方法生成的地图进行了基准测试。研究发现，OpenScene在使用两种编码器时均优于VLMaps，而LSeg在两种建图方法中均优于OpenSeg。