Solely relying on test passing to evaluate Large Language Models (LLMs) for code synthesis may result in unfair assessment or promoting models with data leakage. As an alternative, we introduce CodeMind, a framework designed to gauge the code reasoning abilities of LLMs. CodeMind currently supports three code reasoning tasks: Independent Execution Reasoning (IER), Dependent Execution Reasoning (DER), and Specification Reasoning (SR). The first two evaluate models to predict the execution output of an arbitrary code or code the model could correctly synthesize. The third one evaluates the extent to which LLMs implement the specified expected behavior. Our extensive evaluation of nine LLMs across five benchmarks in two different programming languages using CodeMind shows that LLMs fairly understand control flow constructs and, in general, are capable of reasoning how inputs evolve to output, specifically for simple programs and the ones they can correctly synthesize. However, their performance drops for code with higher complexity, non-trivial logical and arithmetic operators, non-primitive types, and API calls. Furthermore, we observe that, while correlated, specification reasoning (essential for code synthesis) does not imply execution reasoning (essential for broader programming tasks such as testing and debugging): ranking LLMs based on test passing can be different compared to code reasoning.

翻译：仅依赖测试通过率来评估用于代码合成的大型语言模型（LLMs）可能导致不公平的评价，或助长存在数据泄露问题的模型。为此，我们提出CodeMind——一种用于衡量LLMs代码推理能力的框架。CodeMind当前支持三种代码推理任务：独立执行推理（IER）、依赖执行推理（DER）和规约推理（SR）。前两项任务评估模型预测任意代码（或模型能正确合成的代码）执行输出的能力；第三项任务则衡量LLMs实现指定预期行为的程度。我们使用CodeMind对九种LLMs在五个跨两种编程语言的基准测试中进行广泛评估，结果表明：LLMs能较好地理解控制流结构，并普遍具备从输入到输出的推理能力（尤其针对简单程序及其能正确合成的程序）。然而，当代码涉及更高复杂度、非平凡逻辑与算术运算符、非原始类型及API调用时，其性能明显下降。此外，我们观察到，尽管存在相关性，但规约推理（代码合成的关键）并不等同于执行推理（如测试与调试等更广泛编程任务的关键）：基于测试通过率的LLMs排名可能与代码推理能力排名存在差异。