The goal of program synthesis, or code generation, is to generate executable code based on given descriptions. Recently, there has been an increasing number of studies employing reinforcement learning (RL) to improve the performance of large language models (LLMs) for code. However, current representative works either rely solely on offline frameworks, limiting the exploration of new sample spaces, or fall short in the utilization of unit test signals, not accounting for specific error locations within the code. To address these issues, we propose RLTF, i.e., Reinforcement Learning from Unit Test Feedback, a novel online RL framework with unit test feedback of multi-granularity for refining code LLMs. Our approach generates data in real-time during training and simultaneously utilizes fine-grained feedback signals to guide the model towards producing higher-quality code. Extensive experiments show that RLTF achieves state-of-the-art performance on the APPS and the MBPP benchmarks. Our code is available at: https://github.com/Zyq-scut/RLTF.

翻译：程序合成（即代码生成）的目标是根据给定描述生成可执行代码。近期，越来越多的研究采用强化学习来提升大型语言模型在代码生成任务中的表现。然而，现有代表性工作要么仅依赖离线框架，限制了对新样本空间的探索，要么未能充分利用单元测试信号，未对代码中具体错误位置进行建模。为解决这些问题，我们提出RLTF（Reinforcement Learning from Unit Test Feedback），一种新颖的在线强化学习框架，通过多粒度单元测试反馈来优化代码语言模型。该方法在训练过程中实时生成数据，同时利用细粒度反馈信号引导模型生成更高质量的代码。大量实验表明，RLTF在APPS和MBPP基准测试中达到了最先进水平。我们的代码开源地址为：https://github.com/Zyq-scut/RLTF。