Chain-of-thought (CoT) has emerged as a groundbreaking tool in NLP, notably for its efficacy in complex reasoning tasks, such as mathematical proofs. However, its application in code generation faces a distinct challenge, i.e., although the code generated with CoT reasoning is logically correct, it faces the problem of syntax error (e.g., invalid syntax error report) during code execution, which causes the CoT result's pass@1 in HumanEval even lower than the zero-shot result. In this paper, we present Code Chain-of-Thought (CodeCoT) that integrates CoT with a self-examination process for code generation. CodeCoT begins with the LLMs using CoT for initial code development to ensure the generated code follows the correct logic flow. Then, CodeCoT will generate test cases to validate whether the code has syntax errors during the execution. CodeCoT then employs a self-examination phase, in which the generated code is executed against these test cases in the local environment. If the local environment raises error information (e.g., invalid syntax error), CodeCoT will iteratively refine the code based on the feedback information. Within this loop, CodeCoT can make sure their generated codes not only follow the logic flow of the code description, but the syntax error will also be addressed with the self-examination process. Our evaluation results reveal that CodeCoT improves the effectiveness of code generation. For example, CodeCoT increases pass@1 from 75.6% to 79.3% for the HumanEval dataset.

翻译：[translated abstract in Chinese] 链式思维推理（CoT）已成为自然语言处理领域的突破性工具，尤其在数学证明等复杂推理任务中成效显著。然而，其在代码生成领域的应用面临独特挑战：尽管通过CoT推理生成的代码逻辑正确，但在代码执行过程中仍会出现语法错误（例如无效语法错误报告），导致CoT结果的HumanEval数据集pass@1指标甚至低于零样本结果。本文提出代码链式思维推理（CodeCoT），该方法将CoT与自检过程相结合用于代码生成。CodeCoT首先利用LLM通过CoT进行初始代码开发，确保生成的代码遵循正确的逻辑流程；随后生成测试用例以验证代码执行是否存在语法错误。接着进入自检阶段：在本地环境中将生成的代码对测试用例进行执行。当本地环境反馈错误信息（如无效语法错误）时，CodeCoT将基于反馈信息迭代优化代码。在此循环中，CodeCoT既能保证生成的代码遵循代码描述的逻辑流程，又能通过自检过程解决语法错误问题。评估结果表明，CodeCoT显著提升了代码生成的有效性，例如在HumanEval数据集上，pass@1指标从75.6%提升至79.3%。