Complex reasoning ability is one of the most important features of current LLMs, which has also been leveraged to play an integral role in complex decision-making tasks. Therefore, the investigation into the reasoning capabilities of Large Language Models (LLMs) is critical: numerous benchmarks have been established to assess the reasoning abilities of LLMs. However, current benchmarks are inadequate in offering a rigorous evaluation of the full extent of reasoning abilities that LLMs are capable of achieving. They are also prone to the risk of overfitting, as these benchmarks, being publicly accessible and static, allow models to potentially tailor their responses to specific benchmark metrics, thereby inflating their performance. Addressing these limitations, our research introduces a new benchmark, named NPHardEval. This benchmark is designed to evaluate the reasoning abilities of LLMs across a broad spectrum of 900 algorithmic questions, extending up to the NP-Hard complexity class. These questions are meticulously chosen to represent a wide range of complexity class below the NP-hard complexity class, offering a rigorous measure of the reasoning ability of LLMs. Through this study, we shed light on the current state of reasoning in LLMs, providing an objective and rigorous perspective through the comparison of LLMs' performance across complex classes. Moreover, this benchmark is designed with a dynamic update mechanism, where the datapoints are refreshed on a monthly basis. Such regular updates play a crucial role in mitigating the risk of LLMs overfitting to the benchmark, promoting a more accurate and reliable assessment of their reasoning capabilities. The benchmark dataset and code of NPHardEval are available at https://github.com/casmlab/NPHardEval.

翻译：复杂推理能力是当前大语言模型（LLMs）最重要的特征之一，并已在复杂决策任务中发挥关键作用。因此，探究LLMs的推理能力至关重要：已有大量基准被建立以评估LLMs的推理能力。然而，现有基准难以对LLMs所能达到的完整推理能力提供严格评估。此外，由于这些基准公开且静态，模型可能针对特定基准指标调整输出以虚增性能，因此存在过拟合风险。为应对这些局限，本研究提出了名为NPHardEval的新基准。该基准通过涵盖NP-Hard复杂性类别的900个算法问题，全面评估LLMs的推理能力。这些问题经过精心筛选，代表NP-hard复杂性类以下广泛的复杂性类别，为衡量LLMs推理能力提供了严格标准。通过本研究，我们揭示了LLMs当前推理能力的现状，并通过比较LLMs在不同复杂性类上的表现，提供了客观且严谨的视角。此外，该基准设计了动态更新机制，数据点每月刷新。这种定期更新对于缓解LLMs对基准的过拟合风险至关重要，有助于更准确可靠地评估其推理能力。NPHardEval的基准数据集与代码已开源（https://github.com/casmlab/NPHardEval）。