Large Language Models (LLMs) have already become quite proficient at solving simpler programming tasks like those in HumanEval or MBPP benchmarks. However, solving more complex and competitive programming tasks is still quite challenging for these models - possibly due to their tendency to generate solutions as monolithic code blocks instead of decomposing them into logical sub-tasks and sub-modules. On the other hand, experienced programmers instinctively write modularized code with abstraction for solving complex tasks, often reusing previously developed modules. To address this gap, we propose CodeChain, a novel framework for inference that elicits modularized code generation through a chain of self-revisions, each being guided by some representative sub-modules generated in previous iterations. Concretely, CodeChain first instructs the LLM to generate modularized codes through chain-of-thought prompting. Then it applies a chain of self-revisions by iterating the two steps: 1) extracting and clustering the generated sub-modules and selecting the cluster representatives as the more generic and re-usable implementations, and 2) augmenting the original chain-of-thought prompt with these selected module-implementations and instructing the LLM to re-generate new modularized solutions. We find that by naturally encouraging the LLM to reuse the previously developed and verified sub-modules, CodeChain can significantly boost both modularity as well as correctness of the generated solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on CodeContests. It is shown to be effective on both OpenAI LLMs as well as open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation studies with different methods of prompting, number of clusters, model sizes, program qualities, etc., to provide useful insights that underpin CodeChain's success.

翻译：大规模语言模型在解决HumanEval或MBPP基准测试等简单编程任务时已展现出较强能力。然而，这类模型在处理更复杂、更具竞争性的编程任务时仍面临显著挑战——这可能是由于其倾向于将解决方案生成为单一代码块，而非将其分解为逻辑子任务与子模块。相比之下，经验丰富的程序员在解决复杂任务时本能地采用模块化编码并构建抽象层，且常复用先前开发的模块。为弥合这一差距，我们提出CodeChain——一种通过自修正链引导模块化代码生成的新型推理框架，其每次修正均由先前迭代中生成的代表性子模块驱动。具体而言，CodeChain首先通过思维链提示引导大语言模型生成模块化代码；随后通过迭代两个步骤实现自修正链：(1)提取并聚类生成的子模块，选择更具通用性和可复用性的聚类代表作为实现方案；(2)将这些选定的模块实现增强至原始思维链提示中，指导大语言模型重新生成新的模块化解法。我们发现，通过自然激励大语言模型复用先前开发并验证的子模块，CodeChain能显著提升生成代码的模块化程度与正确性，在APPS和CodeContests基准上分别实现35%和76%的pass@1相对提升。该方法在OpenAI大语言模型及WizardCoder等开源大语言模型上均有效。通过采用不同提示方法、聚类数量、模型规模、程序质量等参数的全面消融研究，本文为CodeChain的成功提供了关键见解。