Recent works show that the largest of the large language models (LLMs) can solve many simple reasoning tasks expressed in natural language, without any/much supervision. But, can they also solve challenging first-order combinatorial reasoning problems, such as graph coloring, knapsack and cryptarithmetic? To answer this question, we present PuzzleBench, a dataset of 31 such challenging problems along with a few solved instances for each problem. These problems are all first order, i.e., they can be instantiated with problem instances of varying sizes, and most of them are NP-hard, requiring several reasoning steps to reach the solution. We first observe that LLMs, even when aided by symbolic solvers, perform rather poorly on our dataset. In response, we propose a new approach, Puzzle-LM, which combines LLMs with both symbolic solvers and program interpreters, along with feedback from solved examples, to achieve huge performance gains. Our extensive experimentation and analyses offer new insights into the reasoning abilities and limitations of present-day LLMs.

翻译：近期研究表明，规模最大的大型语言模型（LLMs）能够解决许多以自然语言表达的简单推理任务，几乎无需或完全无需监督。但这类模型能否解决诸如图着色、背包问题和密码算术等具有挑战性的一阶组合推理问题？为回答这一问题，我们提出PuzzleBench数据集，其中包含31个此类挑战性问题及每个问题对应的若干已求解实例。这些问题均为一阶问题，即可通过不同规模的实例进行实例化，且大多数属于NP难问题，需经多步推理才能得出解。我们首先观察到，即使借助符号求解器，LLMs在该数据集上的表现仍相当不佳。为此，我们提出新方法Puzzle-LM，该方法将LLMs与符号求解器及程序解释器相结合，并利用已求解示例的反馈，实现了显著性能提升。通过广泛的实验与分析，我们揭示了当前LLMs在推理能力方面的优势与局限性。