This paper introduces AQA-Bench, a novel benchmark to assess the sequential reasoning capabilities of large language models (LLMs) in algorithmic contexts, such as depth-first search (DFS). The key feature of our evaluation benchmark lies in its interactive evaluation protocol -- for example, in DFS, the availability of each node's connected edge is contingent upon the model's traversal to that node, thereby necessitating the LLM's ability to effectively remember visited nodes and strategize subsequent moves. We comprehensively build AQA-Bench with three different algorithms, namely binary search, depth-first search, and breadth-first search, and to evaluate the sequential reasoning ability of 12 different LLMs. Our investigations reveal several interesting findings: (1) Closed-source models like GPT-4 and Gemini generally show strong sequential reasoning ability, significantly outperforming open-source LLMs. (2) Naively providing interactive examples may inadvertently hurt few-shot performance. (3) A very limited number of predecessor steps following the optimal policy can substantially boost small models' performance. (4) The scaling correlation between performance and model size is not always significant, sometimes even showcasing an inverse trend. We hope our study can catalyze future work on advancing the understanding and enhancement of LLMs' capabilities in sequential reasoning. The code is available at https://github.com/UCSC-VLAA/AQA-Bench.

翻译：本文提出AQA-Bench，一个新颖的基准测试，用于评估大语言模型（LLMs）在算法情境中的序列推理能力，例如深度优先搜索（DFS）。该评估基准的核心特征在于其交互式评估协议——以DFS为例，每个节点连接边的可用性取决于模型是否已遍历至该节点，从而要求LLM能够有效记忆已访问节点并规划后续移动。我们全面构建了包含三种不同算法（二分搜索、深度优先搜索和广度优先搜索）的AQA-Bench，并评估了12种不同LLMs的序列推理能力。研究揭示了若干有趣发现：（1）闭源模型如GPT-4和Gemini通常展现出强大的序列推理能力，显著优于开源LLMs。（2）简单提供交互式示例可能意外损害小样本性能。（3）遵循最优策略的极少量前驱步骤能显著提升小模型的性能。（4）性能与模型规模之间的缩放关联并非始终显著，有时甚至呈现反趋势。我们期望本研究能够推动未来关于理解和增强LLMs序列推理能力的工作。代码已开源，链接为https://github.com/UCSC-VLAA/AQA-Bench。