The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.

翻译：利用在大规模代码语料库上预训练的编程语言模型来自动化软件工程流程，已在代码补全、代码翻译和程序合成等多种代码生成任务中展现出显著潜力。然而，当前方法主要依赖于从文本生成领域借用的监督微调目标，忽略了代码特有的序列级特性，包括但不限于可编译性以及语法和功能正确性。为解决这一局限，我们提出了PPOCoder——一种新型代码生成框架，它将预训练编程语言模型与近端策略优化（一种广泛运用的深度强化学习技术）协同结合。通过利用代码执行与结构对齐提供的不可微反馈，PPOCoder将外部代码特定知识无缝整合至模型优化过程。值得强调的是，PPOCoder是一个任务无关且模型无关的框架，可适用于不同代码生成任务和编程语言。在三个代码生成任务上的大量实验表明，相较于当前最先进方法，我们的方法在多种编程语言的编译成功率和功能正确性上均实现了显著提升，从而验证了其有效性。