Complex reasoning ability is one of the most important features of current LLMs, which has also been leveraged to play an integral role in complex decision-making tasks. Therefore, the investigation into the reasoning capabilities of Large Language Models (LLMs) is critical: numerous benchmarks have been established to assess the reasoning abilities of LLMs. However, current benchmarks are inadequate in offering a rigorous evaluation of the full extent of reasoning abilities that LLMs are capable of achieving. They are also prone to the risk of overfitting, as these benchmarks, being publicly accessible and static, allow models to potentially tailor their responses to specific benchmark metrics, thereby inflating their performance. Addressing these limitations, our research introduces a new benchmark, named NPHardEval. This benchmark is designed to evaluate the reasoning abilities of LLMs across a broad spectrum of 900 algorithmic questions, extending up to the NP-Hard complexity class. These questions are meticulously chosen to represent a wide range of complexity class below the NP-hard complexity class, offering a rigorous measure of the reasoning ability of LLMs. Through this study, we shed light on the current state of reasoning in LLMs, providing an objective and rigorous perspective through the comparison of LLMs' performance across complex classes. Moreover, this benchmark is designed with a dynamic update mechanism, where the datapoints are refreshed on a monthly basis. Such regular updates play a crucial role in mitigating the risk of LLMs overfitting to the benchmark, promoting a more accurate and reliable assessment of their reasoning capabilities. The benchmark dataset and code of NPHardEval are available at https://github.com/casmlab/NPHardEval.

翻译：复杂推理能力是当前大型语言模型（LLMs）最重要的特性之一，它也在复杂决策任务中发挥着关键作用。因此，探究LLMs的推理能力至关重要：目前已建立了众多基准来评估LLMs的推理能力。然而，现有基准在全面、严格评估LLMs所能达到的推理能力方面仍显不足。此外，这些基准因公开、静态而存在过拟合风险——模型可能针对特定基准指标定制回答，从而虚高其性能。为应对这些局限，本研究提出了名为NPHardEval的新基准。该基准旨在通过涵盖NP-困难复杂度类别的900个算法问题，全面评估LLMs的推理能力。这些精心设计的问题代表了NP-困难复杂度类别以下广泛复杂度类别的典型范例，为测度LLMs推理能力提供了严格手段。通过本研究，我们揭示了LLMs推理能力的现状，并通过对比LLMs在不同复杂度类别上的表现，提供了客观、严谨的视角。此外，该基准设计了动态更新机制，数据点每月定期刷新。这种定期更新能有效降低LLMs对基准过拟合的风险，促进对其推理能力的更准确、可靠评估。NPHardEval的基准数据集和代码已在https://github.com/casmlab/NPHardEval 公开。