Large Language Models (LLMs) pre-trained on code have recently emerged as the dominant approach to program synthesis. However, the code that these models produce can violate basic language-level invariants, leading to lower performance in downstream tasks. We address this issue through an approach, called RLCF, that further trains a pre-trained LLM using feedback from a code compiler. RLCF views the LLM as an RL agent that generates code step by step and receives: (i) compiler-derived feedback on whether the code it generates passes a set of correctness checks; and (ii) feedback from a different LLM on whether the generated code is similar to a set of reference programs in the training corpus. Together, these feedback mechanisms help the generated code remain within the target distribution while passing all static correctness checks. RLCF is model- and language-agnostic. We empirically evaluate it on the MBJP and MathQA tasks for Java. Our experiments show that RLCF significantly raises the odds that an LLM-generated program compiles, is executable, and produces the right output on tests, often allowing LLMs to match the performance of 2x-8x larger LLMs.

翻译：大规模预训练于代码的大型语言模型（LLMs）最近已成为程序合成的主流方法。然而，这些模型生成的代码可能违反基本的语言级不变量，导致下游任务性能下降。我们通过一种名为RLCF的方法解决这一问题，该方法利用代码编译器的反馈对预训练的LLM进行进一步训练。RLCF将LLM视为一个逐步生成代码的强化学习智能体，并接收：（i）编译器对其生成代码是否通过一组正确性检查的反馈；（ii）另一个LLM对其生成代码是否与训练语料库中一组参考程序相似的反馈。这些反馈机制共同帮助生成的代码在保持目标分布内的同时，通过所有静态正确性检查。RLCF是模型和语言无关的。我们在Java的MBJP和MathQA任务上进行了实证评估。实验结果表明，RLCF显著提高了LLM生成程序通过编译、可执行并在测试中产生正确输出的概率，通常使LLM能够与规模大2倍至8倍的LLM性能相匹配。