Complex reasoning ability is one of the most important features of current LLMs, which has also been leveraged to play an integral role in complex decision-making tasks. Therefore, the investigation into the reasoning capabilities of Large Language Models (LLMs) is critical: numerous benchmarks have been established to assess the reasoning abilities of LLMs. However, current benchmarks are inadequate in offering a rigorous evaluation of the full extent of reasoning abilities that LLMs are capable of achieving. They are also prone to the risk of overfitting, as these benchmarks, being publicly accessible and static, allow models to potentially tailor their responses to specific benchmark metrics, thereby inflating their performance. Addressing these limitations, our research introduces a new benchmark, named NPHardEval. This benchmark is designed to evaluate the reasoning abilities of LLMs across a broad spectrum of 900 algorithmic questions, extending up to the NP-Hard complexity class. These questions are meticulously chosen to represent a wide range of complexity class below the NP-hard complexity class, offering a rigorous measure of the reasoning ability of LLMs. Through this study, we shed light on the current state of reasoning in LLMs, providing an objective and rigorous perspective through the comparison of LLMs' performance across complex classes. Moreover, this benchmark is designed with a dynamic update mechanism, where the datapoints are refreshed on a monthly basis. Such regular updates play a crucial role in mitigating the risk of LLMs overfitting to the benchmark, promoting a more accurate and reliable assessment of their reasoning capabilities. The benchmark dataset and code of NPHardEval are available at https://github.com/casmlab/NPHardEval.

翻译：复杂推理能力是当前大语言模型（LLMs）最重要的特征之一，并已在复杂决策任务中发挥关键作用。因此，探究LLMs的推理能力至关重要：目前已建立众多基准来评估LLMs的推理能力。然而，现有基准难以对LLMs可能达到的全部推理能力提供严格评估，且因公开静态的特性存在过拟合风险——模型可能针对特定基准指标调整回答，从而虚增性能。针对这些局限，本研究提出了名为NPHardEval的新基准。该基准通过覆盖NP-Hard复杂度类的900个算法问题，系统评估LLMs的推理能力。这些问题经过精心筛选，全面涵盖NP-hard复杂度类以下的各类复杂度层级，为LLMs的推理能力提供严格度量。通过本研究，我们揭示了LLMs当前推理能力的现状，通过跨复杂度类的性能对比提供了客观严谨的视角。此外，该基准配备动态更新机制，数据点每月刷新。这种定期更新对缓解LLMs过拟合基准的风险至关重要，有助于更准确可靠地评估其推理能力。NPHardEval的基准数据集和代码可从https://github.com/casmlab/NPHardEval获取。