Code generation tools driven by artificial intelligence have recently become more popular due to advancements in deep learning and natural language processing that have increased their capabilities. The proliferation of these tools may be a double-edged sword because while they can increase developer productivity by making it easier to write code, research has shown that they can also generate insecure code. In this paper, we perform a user-centered evaluation GitHub's Copilot to better understand its strengths and weaknesses with respect to code security. We conduct a user study where participants solve programming problems (with and without Copilot assistance) that have potentially vulnerable solutions. The main goal of the user study is to determine how the use of Copilot affects participants' security performance. In our set of participants (n=25), we find that access to Copilot accompanies a more secure solution when tackling harder problems. For the easier problem, we observe no effect of Copilot access on the security of solutions. We also observe no disproportionate impact of Copilot use on particular kinds of vulnerabilities. Our results indicate that there are potential security benefits to using Copilot, but more research is warranted on the effects of the use of code generation tools on technically complex problems with security requirements.

翻译：由人工智能驱动的代码生成工具，因深度学习与自然语言处理技术的进步而功能增强，近年来日益普及。这些工具的广泛使用可能是一把双刃剑：一方面，它们通过简化代码编写流程提升开发者效率；另一方面，研究显示它们可能生成不安全的代码。本文对GitHub的Copilot进行以用户为中心的安全评估，以深入理解其在代码安全方面的优势与不足。我们开展了一项用户研究，受试者需解决存在潜在脆弱性编程问题（分别在有/无Copilot辅助条件下）。该用户研究的主要目标是确定Copilot的使用如何影响受试者的安全性能。在参与研究的25名受试者中，我们发现：当解决较难问题时，使用Copilot伴随更安全的解决方案；对于较简单问题，未观察到Copilot访问对方案安全性的影响；同时未发现Copilot使用对特定类型漏洞存在显著差异。我们的结果表明，使用Copilot具有潜在安全优势，但关于代码生成工具对含安全需求的技术复杂问题的影响，仍需更多研究加以验证。