AI assistants for coding are on the rise. However one of the reasons developers and companies avoid harnessing their full potential is the questionable security of the generated code. This paper first reviews the current state-of-the-art and identifies areas for improvement on this issue. Then, we propose a systematic approach based on prompt-altering methods to achieve better code security of (even proprietary black-box) AI-based code generators such as GitHub Copilot, while minimizing the complexity of the application from the user point-of-view, the computational resources, and operational costs. In sum, we propose and evaluate three prompt altering methods: (1) scenario-specific, (2) iterative, and (3) general clause, while we discuss their combination. Contrary to the audit of code security, the latter two of the proposed methods require no expert knowledge from the user. We assess the effectiveness of the proposed methods on the GitHub Copilot using the OpenVPN project in realistic scenarios, and we demonstrate that the proposed methods reduce the number of insecure generated code samples by up to 16\% and increase the number of secure code by up to 8\%. Since our approach does not require access to the internals of the AI models, it can be in general applied to any AI-based code synthesizer, not only GitHub Copilot.
翻译:AI编码助手正日益普及,然而开发者和企业未能充分利用其潜能的原因之一在于所生成代码的安全性存疑。本文首先回顾了该领域的现有研究进展,并指出需要改进的方向。随后,我们提出了一种基于提示修改方法的系统化方案,用于提升(即使是专有黑盒)AI代码生成器(如GitHub Copilot)的代码安全性,同时从用户视角最小化应用复杂度、计算资源与运行成本。具体而言,我们提出并评估了三种提示修改方法:(1)场景特定法,(2)迭代法,(3)通用条款法,并探讨了它们的组合应用。与代码安全审计不同,后两种方法无需用户具备专业知识。我们以OpenVPN项目为真实场景,在GitHub Copilot上评估了所提方法的有效性,结果表明:所提方法可使不安全生成代码样本数量最多减少16%,安全代码数量最多增加8%。由于该方法无需访问AI模型的内部机制,因此可普遍适用于任何基于AI的代码合成器,而不仅限于GitHub Copilot。