Code generation tools driven by artificial intelligence have recently become more popular due to advancements in deep learning and natural language processing that have increased their capabilities. The proliferation of these tools may be a double-edged sword because while they can increase developer productivity by making it easier to write code, research has shown that they can also generate insecure code. In this paper, we perform a user-centered evaluation GitHub's Copilot to better understand its strengths and weaknesses with respect to code security. We conduct a user study where participants solve programming problems, which have potentially vulnerable solutions, with and without Copilot assistance. The main goal of the user study is to determine how the use of Copilot affects participants' security performance. In our set of participants (n=25), we find that access to Copilot accompanies a more secure solution when tackling harder problems. For the easier problem, we observe no effect of Copilot access on the security of solutions. We also observe no disproportionate impact of Copilot use on particular kinds of vulnerabilities.

翻译：人工智能驱动的代码生成工具因深度学习和自然语言处理技术的进步而能力大增，近年来变得日益普及。这些工具的广泛使用可能是一把双刃剑，因为它们虽然能通过简化代码编写提升开发效率，但研究表明它们也可能生成不安全的代码。本文对 GitHub 的 Copilot 进行了以用户为中心的评估，以更深入地理解其在代码安全方面的优势与缺陷。我们开展了一项用户研究，参与者需解决具有潜在脆弱性解决方案的编程问题，部分参与者使用 Copilot 辅助，其余则不使用。该用户研究的主要目标是确定 Copilot 的使用如何影响参与者的安全性能。在我们的参与者样本中（n=25），我们发现使用 Copilot 在解决较难问题时能带来更安全的解决方案。对于较简单的问题，我们未观察到 Copilot 的使用对解决方案安全性有显著影响。同时，我们也未发现 Copilot 的使用对特定类型的漏洞有不成比例的影响。