The rapid evolution of large language models (LLMs) holds promise for reforming the methodology of spatio-temporal data mining. However, current works for evaluating the spatio-temporal understanding capability of LLMs are somewhat limited and biased. These works either fail to incorporate the latest language models or only focus on assessing the memorized spatio-temporal knowledge. To address this gap, this paper dissects LLMs' capability of spatio-temporal data into four distinct dimensions: knowledge comprehension, spatio-temporal reasoning, accurate computation, and downstream applications. We curate several natural language question-answer tasks for each category and build the benchmark dataset, namely STBench, containing 13 distinct tasks and over 60,000 QA pairs. Moreover, we have assessed the capabilities of 13 LLMs, such as GPT-4o, Gemma and Mistral. Experimental results reveal that existing LLMs show remarkable performance on knowledge comprehension and spatio-temporal reasoning tasks, with potential for further enhancement on other tasks through in-context learning, chain-of-though prompting, and fine-tuning. The code and datasets of STBench are released on https://github.com/LwbXc/STBench.

翻译：大语言模型（LLMs）的快速发展为时空数据挖掘方法的革新带来了希望。然而，当前用于评估LLMs时空理解能力的工作存在一定的局限性和偏差。这些研究要么未能纳入最新的语言模型，要么仅侧重于评估模型记忆的时空知识。为弥补这一空白，本文将LLMs的时空数据处理能力解构为四个不同的维度：知识理解、时空推理、精确计算以及下游应用。我们为每个类别精心设计了若干自然语言问答任务，并构建了基准数据集，即STBench，包含13个不同的任务和超过60,000个问答对。此外，我们评估了包括GPT-4o、Gemma和Mistral在内的13种LLMs的能力。实验结果表明，现有LLMs在知识理解和时空推理任务上表现出色，而通过情境学习、思维链提示和微调，它们在其他任务上仍有进一步提升的潜力。STBench的代码和数据集已发布于https://github.com/LwbXc/STBench。