In recent years, the use of automated source code generation utilizing transformer-based generative models has expanded, and these models can generate functional code according to the requirements of the developers. However, recent research revealed that these automatically generated source codes can contain vulnerabilities and other quality issues. Despite researchers' and practitioners' attempts to enhance code generation models, retraining and fine-tuning large language models is time-consuming and resource-intensive. Thus, we describe FRANC, a lightweight framework for recommending more secure and high-quality source code derived from transformer-based code generation models. FRANC includes a static filter to make the generated code compilable with heuristics and a quality-aware ranker to sort the code snippets based on a quality score. Moreover, the framework uses prompt engineering to fix persistent quality issues. We evaluated the framework with five Python and Java code generation models and six prompt datasets, including a newly created one in this work (SOEval). The static filter improves 9% to 46% Java suggestions and 10% to 43% Python suggestions regarding compilability. The average improvement over the NDCG@10 score for the ranking system is 0.0763, and the repairing techniques repair the highest 80% of prompts. FRANC takes, on average, 1.98 seconds for Java; for Python, it takes 0.08 seconds.
翻译:近年来,基于Transformer的生成模型在自动化源代码生成领域的应用日益广泛,这类模型能够根据开发者需求生成功能性代码。然而最新研究表明,这些自动生成的源代码可能存在安全漏洞及其他质量问题。尽管研究人员与从业者尝试改进代码生成模型,但重新训练与微调大语言模型仍需耗费大量时间与资源。为此,我们提出FRANC框架——一个轻量级框架,用于从基于Transformer的代码生成模型中推荐更安全、更高质量的源代码。FRANC包含静态过滤器,通过启发式策略确保生成代码的可编译性;并配备质量感知排序器,基于质量评分对代码片段进行排序。此外,该框架运用提示工程修复持续性质量问题。我们通过五个Python与Java代码生成模型及六个提示数据集(含本研究新构建的SOEval)对框架进行评估。静态过滤器在可编译性方面,将Java建议提升9%-46%,Python建议提升10%-43%。排序系统的NDCG@10平均得分提升0.0763,修复技术可修复最高80%的提示。FRANC处理Java代码平均耗时1.98秒,处理Python代码平均耗时0.08秒。