Large-scale code generation models such as Codex and CodeT5 have achieved impressive performance. However, libraries are upgraded or deprecated very frequently and re-training large-scale language models is computationally expensive. Therefore, Continual Learning (CL) is an important aspect that remains underexplored in the code domain. In this paper, we introduce a benchmark called CodeTask-CL that covers a wide range of tasks, including code generation, translation, summarization, and refinement, with different input and output programming languages. Next, on our CodeTask-CL benchmark, we compare popular CL techniques from NLP and Vision domains. We find that effective methods like Prompt Pooling (PP) suffer from catastrophic forgetting due to the unstable training of the prompt selection mechanism caused by stark distribution shifts in coding tasks. We address this issue with our proposed method, Prompt Pooling with Teacher Forcing (PP-TF), that stabilizes training by enforcing constraints on the prompt selection mechanism and leads to a 21.54% improvement over Prompt Pooling. Along with the benchmark, we establish a training pipeline that can be used for CL on code models, which we believe can motivate further development of CL methods for code models. Our code is available at https://github.com/amazon-science/codetaskcl-pptf

翻译：大规模代码生成模型（如Codex和CodeT5）已取得显著性能。然而，代码库的更新或废弃非常频繁，而重新训练大规模语言模型的计算成本高昂。因此，持续学习（CL）成为代码领域尚待深入探索的重要方向。本文提出了一个名为CodeTask-CL的基准测试，涵盖代码生成、翻译、摘要和优化等多样化任务，涉及不同的输入与输出编程语言。随后，我们在CodeTask-CL基准上比较了来自自然语言处理（NLP）和计算机视觉领域的流行CL技术。研究发现，提示池化（PP）等有效方法因编码任务中显著的数据分布偏移导致提示选择机制训练不稳定，从而遭受灾难性遗忘。针对这一问题，我们提出了一种新方法——带教师强制的提示池化（PP-TF），通过对提示选择机制施加约束来稳定训练，相较于PP方法取得了21.54%的性能提升。除基准测试外，我们还建立了一个可用于代码模型持续学习的训练流程，相信这将推动代码模型持续学习方法的进一步发展。我们的代码已开源：https://github.com/amazon-science/codetaskcl-pptf