Temporal Reasoning (TR) is a critical ability for LLMs to understand and reason over temporal information and relationships between events. To study the TR ability in LLMs, prior works provide different ways for evaluating various aspects of TR ability. In this work, we propose an alternative perspective for evaluating TR ability by leveraging Linear Temporal Logic (LTL), and develop a pipeline to automatically synthesize challenges for assessing the TR ability of LLMs. Based on this pipeline, we construct a dataset, namely LTLBench, consisting of $2000$ TR challenges, and benchmark 12 LLMs across 5 different methods. Furthermore, we conduct additional experiments to investigate the impact of increasing the number of formula operators and events on both LLM performance and the complexity of TR problems. We also perform qualitative analyses of their reasoning processes and the effects of varying the number of events and formula operators, which reveal 3 main issues in their temporal reasoning processes and the unexpected performance changes observed as problem complexity increases. We expect this work to provide valuable insights into the TR ability of LLMs.

翻译：时序推理（TR）是大型语言模型（LLM）理解并推演时间信息及事件间关系的关键能力。为研究LLM的TR能力，先前工作提出了多种评估TR能力不同维度的方案。本工作提出一种基于线性时序逻辑（LTL）的TR能力评估新视角，并开发了自动生成TR能力测试难题的流程框架。基于该框架，我们构建了包含$2000$个TR测试难题的数据集LTLBench，并采用5种不同方法对12个LLM进行了基准测试。此外，我们通过扩展实验探究了公式算子数量与事件数量增加对LLM性能及TR问题复杂性的影响。通过对模型推理过程以及事件数量与公式算子数量变化效应的定性分析，我们揭示了其时序推理过程中存在的3类主要问题，以及问题复杂度提升时出现的非预期性能变化现象。本研究期望为深入理解LLM的时序推理能力提供有价值的见解。