Large language models (LLMs) have brought significant advancements to code generation, benefiting both novice and experienced developers. However, their training using unsanitized data from open-source repositories, like GitHub, introduces the risk of inadvertently propagating security vulnerabilities. To effectively mitigate this concern, this paper presents a comprehensive study focused on evaluating and enhancing code LLMs from a software security perspective. We introduce SecuCoGen\footnote{SecuCoGen has been uploaded as supplemental material and will be made publicly available after publication.}, a meticulously curated dataset targeting 21 critical vulnerability types. SecuCoGen comprises 180 samples and serves as the foundation for conducting experiments on three crucial code-related tasks: code generation, code repair and vulnerability classification, with a strong emphasis on security. Our experimental results reveal that existing models often overlook security concerns during code generation, leading to the generation of vulnerable code. To address this, we propose effective approaches to mitigate the security vulnerabilities and enhance the overall robustness of code generated by LLMs. Moreover, our study identifies weaknesses in existing models' ability to repair vulnerable code, even when provided with vulnerability information. Additionally, certain vulnerability types pose challenges for the models, hindering their performance in vulnerability classification. Based on these findings, we believe our study will have a positive impact on the software engineering community, inspiring the development of improved methods for training and utilizing LLMs, thereby leading to safer and more trustworthy model deployment.

翻译：大语言模型为代码生成带来了重大进展，惠及了新手和经验丰富的开发者。然而，这些模型使用来自开源代码仓库（如GitHub）的未经清洗的数据进行训练，带来了无意中传播安全漏洞的风险。为有效缓解这一问题，本文提出了一项综合研究，从软件安全角度评估和增强代码大语言模型。我们引入了SecuCoGen（已作为补充材料上传，将在发表后公开），这是一个针对21种关键漏洞类型精心整理的数据集。SecuCoGen包含180个样本，为在三个关键代码相关任务（代码生成、代码修复和漏洞分类，重点关注安全）上开展实验奠定了基础。我们的实验结果表明，现有模型在代码生成过程中常忽视安全问题，导致生成含有漏洞的代码。为此，我们提出了缓解安全漏洞并增强大语言模型生成代码整体鲁棒性的有效方法。此外，我们的研究揭示了现有模型在修复漏洞代码方面的能力缺陷——即使提供了漏洞信息，其修复能力仍显不足。同时，某些漏洞类型对模型构成挑战，影响了它们在漏洞分类中的性能。基于这些发现，我们相信本研究将对软件工程界产生积极影响，推动训练和使用大语言模型的改进方法的发展，从而实现更安全、更可信的模型部署。