With the advances in machine learning, there is a growing interest in AI-enabled tools for autocompleting source code. GitHub Copilot, also referred to as the "AI Pair Programmer", has been trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch on June 2021. However, little effort has been devoted to understanding the practices and challenges of using Copilot in programming with auto-completed source code. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. More specifically, we searched and manually collected 169 SO posts and 655 GitHub discussions related to the usage of Copilot. We identified the programming languages, IDEs, technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. The results show that when practitioners use Copilot: (1) The major programming languages used with Copilot are JavaScript and Python, (2) the main IDE used with Copilot is Visual Studio Code, (3) the most common used technology with Copilot is Node.js, (4) the leading function implemented by Copilot is data processing, (5) the significant benefit of using Copilot is useful code generation, and (6) the main limitation encountered by practitioners when using Copilot is difficulty of integration. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it. Our study provides empirically grounded foundations and basis for future research on the role of Copilot as an AI pair programmer in software development.
翻译:随着机器学习技术的进步,基于AI的源代码自动补全工具日益受到关注。GitHub Copilot(又称"AI结对程序员")基于数十亿行开源GitHub代码训练而成,自2021年6月发布以来已成为此类工具中应用日益广泛的代表。然而,目前鲜有研究探索使用Copilot进行源代码自动补全编程的实践与挑战。为此,我们通过收集并分析Stack Overflow(SO)与GitHub Discussions数据开展了实证研究。具体而言,我们检索并手工搜集了169篇SO帖子与655条GitHub讨论,内容涉及Copilot的使用实践。我们识别了使用Copilot的编程语言、集成开发环境、技术栈、实现功能、优势、局限与挑战。研究结果表明:当实践者使用Copilot时:(1)主要编程语言为JavaScript与Python;(2)主要集成开发环境为Visual Studio Code;(3)最常见的技术栈为Node.js;(4)实现的核心功能为数据处理;(5)显著优势在于生成有效代码;(6)实践者遇到的主要局限是集成困难。我们的研究表明,使用Copilot犹如双刃剑,开发者在决定是否采纳时需审慎权衡多方面因素。本研究为未来探索Copilot作为AI结对程序员在软件开发中的角色奠定了实证基础。