Code generation tools driven by artificial intelligence have recently become more popular due to advancements in deep learning and natural language processing that have increased their capabilities. The proliferation of these tools may be a double-edged sword because while they can increase developer productivity by making it easier to write code, research has shown that they can also generate insecure code. In this paper, we perform a user-centered evaluation GitHub's Copilot to better understand its strengths and weaknesses with respect to code security. We conduct a user study where participants solve programming problems (with and without Copilot assistance) that have potentially vulnerable solutions. The main goal of the user study is to determine how the use of Copilot affects participants' security performance. In our set of participants (n=25), we find that access to Copilot accompanies a more secure solution when tackling harder problems. For the easier problem, we observe no effect of Copilot access on the security of solutions. We also observe no disproportionate impact of Copilot use on particular kinds of vulnerabilities. Our results indicate that there are potential security benefits to using Copilot, but more research is warranted on the effects of the use of code generation tools on technically complex problems with security requirements.

翻译：基于人工智能的代码生成工具因深度学习和自然语言处理技术的进步而日益普及，其能力也随之增强。这些工具的普及可能是一把双刃剑：一方面，它们通过简化代码编写过程提升了开发者的生产力；另一方面，研究表明它们也可能生成不安全的代码。本文对GitHub的Copilot进行用户中心评估，以更全面地理解其在代码安全方面的优势与不足。我们开展了一项用户研究，要求参与者解决存在潜在漏洞的编程问题（部分使用Copilot辅助，部分不使用）。该用户研究的主要目标是探究Copilot的使用如何影响参与者的安全表现。在我们的参与者群体（n=25）中，我们发现：在解决较困难的问题时，使用Copilot与生成更安全的解决方案相关；而对于较简单的问题，Copilot的使用对解决方案的安全性没有显著影响。此外，我们未观察到Copilot的使用对特定类型漏洞产生不成比例的影响。研究结果表明，使用Copilot可能存在安全效益，但关于代码生成工具对具有安全需求的技术复杂问题的影响，仍需进一步研究。