Mapping and navigation services like Google Maps, Apple Maps, Openstreet Maps, are essential for accessing various location-based data, yet they often struggle to handle natural language geospatial queries. Recent advancements in Large Language Models (LLMs) show promise in question answering (QA), but creating reliable geospatial QA datasets from map services remains challenging. We introduce MapQaTor, a web application that streamlines the creation of reproducible, traceable map-based QA datasets. With its plug-and-play architecture, MapQaTor enables seamless integration with any maps API, allowing users to gather and visualize data from diverse sources with minimal setup. By caching API responses, the platform ensures consistent ground truth, enhancing the reliability of the data even as real-world information evolves. MapQaTor centralizes data retrieval, annotation, and visualization within a single platform, offering a unique opportunity to evaluate the current state of LLM-based geospatial reasoning while advancing their capabilities for improved geospatial understanding. Evaluation metrics show that, MapQaTor speeds up the annotation process by at least 30 times compared to manual methods, underscoring its potential for developing geospatial resources, such as complex map reasoning datasets. The website is live at: https://mapqator.github.io/ and a demo video is available at: https://youtu.be/7_aV9Wmhs6Q.

翻译：诸如Google Maps、Apple Maps、Openstreet Maps等地图与导航服务对于获取各类基于位置的数据至关重要，然而它们往往难以处理自然语言地理空间查询。大型语言模型（LLMs）在问答（QA）领域的最新进展展现出潜力，但从地图服务创建可靠的地理空间QA数据集仍具挑战性。我们推出MapQaTor——一个简化可复现、可追溯的地图问答数据集创建的Web应用程序。凭借其即插即用架构，MapQaTor能够与任何地图API无缝集成，使用户能以最小设置从多样化来源收集和可视化数据。通过缓存API响应，该平台确保了基准真值的一致性，即使在现实世界信息动态变化时也能增强数据的可靠性。MapQaTor将数据检索、标注和可视化集中在一个平台内，为评估当前基于LLM的地理空间推理水平提供了独特机会，同时推动其提升地理空间理解能力。评估指标表明，相较于人工方法，MapQaTor将标注流程加速至少30倍，彰显了其在开发复杂地图推理数据集等地理空间资源方面的潜力。网站访问地址：https://mapqator.github.io/，演示视频详见：https://youtu.be/7_aV9Wmhs6Q。