Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options.

翻译：大型语言模型（LLM）在给定包含中间推理步骤的思维链提示示例时，展现出显著的推理能力。现有基准通过评估数学推理等下流任务的准确率来间接衡量推理能力，然而，尚不清楚这些模型是如何得出答案的，以及它们是否依赖简单的启发式方法而非生成的思维链。为了系统性地探究LLM的推理能力，我们提出了一个名为PrOntoQA的新型合成问答数据集，其中每个示例均基于以一阶逻辑表示的合成世界模型生成。这使得我们能够将生成的思维链解析为符号化证明以进行形式分析。我们对InstructGPT和GPT-3的分析表明，LLM在执行正确的单个演绎步骤方面相当擅长，因此即使在虚构语境下也普遍具备推理能力。然而，它们在证明规划方面存在困难：当存在多个有效的演绎步骤时，它们无法系统性地探索不同的选项。