With the advances in Machine Learning, there is a growing interest in AI-enabled tools for autocompleting source code. GitHub Copilot, also referred to as the "AI Pair Programmer", has trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch on June 2021. However, little effort has been devoted to understand the practices and challenges of using Copilot in programming with auto-completed source code. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. More specifically, we searched and manually collected 169 SO posts and 655 GitHub discussions related to the usage of Copilot. We identified the programming languages, IDEs, technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. The results show that when practitioners use Copilot: (1) The major programming languages used with Copilot are JavaScript and Python, (2) the main IDE used with Copilot is Visual Studio Code, (3) the most common used technology with Copilot is Node.js, (4) the leading function implemented by Copilot is data processing, (5) the significant benefit of using Copilot is useful code generation, and (6) the main limitation encountered by practitioners when using Copilot is difficulty of integration. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it. Our study provides empirically grounded foundations and basis for future research on the role of Copilot as an AI pair programmer in software development.
翻译:随着机器学习的进步,具备源代码自动补全功能的AI工具日益受到关注。GitHub Copilot(又称"AI结对程序员")基于数十亿行开源GitHub代码训练而成,自2021年6月发布以来使用量持续增长。然而,目前鲜有研究探讨开发者在使用自动补全源代码的Copilot进行编程时的实践与挑战。为此,我们通过收集并分析Stack Overflow (SO) 和GitHub Discussions上的数据开展实证研究。具体而言,我们检索并手动收集了169篇SO帖子及655篇与Copilot使用相关的GitHub讨论。我们识别了与Copilot搭配使用的编程语言、集成开发环境、技术栈、实现功能,以及使用Copilot带来的益处、局限与挑战。研究结果表明:当从业者使用Copilot时,(1)主要编程语言为JavaScript和Python,(2)主要集成开发环境为Visual Studio Code,(3)最常搭配的技术为Node.js,(4)Copilot实现的核心功能为数据处理,(5)使用Copilot的显著优势在于生成有效代码,(6)从业者遇到的主要限制是集成困难。我们的研究结果表明,Copilot的使用如同一把双刃剑,要求开发者在决定是否使用时需审慎权衡多方面因素。本研究为后续探讨Copilot作为AI结对程序员在软件开发中的角色奠定了实证基础与依据。