AI-powered code generation models have been developing rapidly, allowing developers to expedite code generation and thus improve their productivity. These models are trained on large corpora of code (primarily sourced from public repositories), which may contain bugs and vulnerabilities. Several concerns have been raised about the security of the code generated by these models. Recent studies have investigated security issues in AI-powered code generation tools such as GitHub Copilot and Amazon CodeWhisperer, revealing several security weaknesses in the code generated by these tools. As these tools evolve, it is expected that they will improve their security protocols to prevent the suggestion of insecure code to developers. This paper replicates the study of Pearce et al., which investigated security weaknesses in Copilot and uncovered several weaknesses in the code suggested by Copilot across diverse scenarios and languages (Python, C and Verilog). Our replication examines Copilot security weaknesses using newer versions of Copilot and CodeQL (the security analysis framework). The replication focused on the presence of security vulnerabilities in Python code. Our results indicate that, even with the improvements in newer versions of Copilot, the percentage of vulnerable code suggestions has reduced from 36.54% to 27.25%. Nonetheless, it remains evident that the model still suggests insecure code.

翻译：基于人工智能的代码生成模型发展迅速，帮助开发者加快代码生成速度，从而提升生产效率。这些模型在大型代码语料库（主要来源于公共代码仓库）上训练，而这些语料库可能包含错误和安全漏洞。有关这些模型生成代码的安全性问题已引发多方关注。近期研究调查了GitHub Copilot和Amazon CodeWhisperer等AI驱动代码生成工具的安全问题，揭示了这些工具生成代码中存在的若干安全弱点。随着这些工具的持续进化，预计其安全协议将得到改进，以防范向开发者推荐不安全代码。本文复制了Pearce等人的研究，该研究调查了Copilot的安全弱点，并揭示了Copilot在不同场景和语言（Python、C和Verilog）中推荐代码的多个安全问题。我们的复制研究使用更新版本的Copilot和CodeQL（安全分析框架）来检验Copilot的安全弱点，重点关注Python代码中存在的安全漏洞。结果表明，尽管新版Copilot有所改进，存在漏洞的代码推荐比例已从36.54%降至27.25%，但模型仍倾向于推荐不安全代码的事实依然明显。