Solely relying on test passing to evaluate Large Language Models (LLMs) for code synthesis may result in unfair assessment or promoting models with data leakage. As an alternative, we introduce CodeMind, a framework designed to gauge the code reasoning abilities of LLMs. CodeMind currently supports three code reasoning tasks: Independent Execution Reasoning (IER), Dependent Execution Reasoning (DER), and Specification Reasoning (SR). The first two evaluate models to predict the execution output of an arbitrary code or code the model could correctly synthesize. The third one evaluates the extent to which LLMs implement the specified expected behavior. Our extensive evaluation of nine LLMs across five benchmarks in two different programming languages using CodeMind shows that LLMs fairly understand control flow constructs and, in general, are capable of reasoning how inputs evolve to output, specifically for simple programs and the ones they can correctly synthesize. However, their performance drops for code with higher complexity, non-trivial logical and arithmetic operators, non-primitive types, and API calls. Furthermore, we observe that, while correlated, specification reasoning (essential for code synthesis) does not imply execution reasoning (essential for broader programming tasks such as testing and debugging): ranking LLMs based on test passing can be different compared to code reasoning.

翻译：仅依赖测试通过来评估大语言模型（LLM）的代码合成能力可能导致不公平评估或助推存在数据泄露的模型。为此，我们提出CodeMind——一个专为衡量LLM代码推理能力而设计的框架。CodeMind当前支持三种代码推理任务：独立执行推理（IER）、依赖执行推理（DER）和规范推理（SR）。前两项任务评估模型预测任意代码或模型可正确合成代码的执行输出能力；第三项则衡量LLM实现指定预期行为的程度。我们使用CodeMind对五种基准测试中的九种LLM（涵盖两种不同编程语言）进行广泛评估，结果表明：LLM能较好理解控制流结构，通常可推演输入如何转化为输出（尤其针对简单程序及其能正确合成的程序）。然而，当代码复杂度升高、涉及非平凡逻辑与算术运算符、非原始类型及API调用时，其性能显著下降。此外，我们发现尽管规范推理（代码合成的核心能力）与执行推理（测试与调试等广义编程任务的核心能力）存在关联，但两者并不等同：基于测试通过率的LLM排名可能不同于基于代码推理能力的排名。