With the advances in machine learning, there is a growing interest in AI-enabled tools for autocompleting source code. GitHub Copilot, also referred to as the "AI Pair Programmer", has been trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch on June 2021. However, little effort has been devoted to understanding the practices and challenges of using Copilot in programming with auto-completed source code. To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions. More specifically, we searched and manually collected 169 SO posts and 655 GitHub discussions related to the usage of Copilot. We identified the programming languages, IDEs, technologies used with Copilot, functions implemented, benefits, limitations, and challenges when using Copilot. The results show that when practitioners use Copilot: (1) The major programming languages used with Copilot are JavaScript and Python, (2) the main IDE used with Copilot is Visual Studio Code, (3) the most common used technology with Copilot is Node.js, (4) the leading function implemented by Copilot is data processing, (5) the significant benefit of using Copilot is useful code generation, and (6) the main limitation encountered by practitioners when using Copilot is difficulty of integration. Our results suggest that using Copilot is like a double-edged sword, which requires developers to carefully consider various aspects when deciding whether or not to use it. Our study provides empirically grounded foundations and basis for future research on the role of Copilot as an AI pair programmer in software development.
翻译:随着机器学习的进步,人们对用于自动补全源代码的人工智能工具的兴趣日益增长。GitHub Copilot,也被称为"AI结对程序员",基于数十亿行开源GitHub代码进行了训练,自2021年6月发布以来已成为日益广泛使用的此类工具之一。然而,目前尚缺乏对使用Copilot进行自动补全编程的实践与挑战的系统性理解。为此,我们通过收集和分析Stack Overflow(SO)与GitHub Discussions的数据开展了实证研究。具体而言,我们搜索并手动收集了169篇SO帖子和655个与Copilot使用相关的GitHub讨论。我们识别了与Copilot配合使用的编程语言、集成开发环境、技术栈,以及Copilot实现的功能、优势、局限和挑战。结果表明,实践者在使用Copilot时:(1)主要编程语言为JavaScript和Python,(2)主要集成开发环境为Visual Studio Code,(3)最常配合使用的技术为Node.js,(4)Copilot实现的首要功能为数据处理,(5)使用Copilot的显著优势在于生成有用代码,(6)实践者遇到的主要局限是集成困难。我们的研究表明,使用Copilot如同双刃剑,需要开发者在决定是否使用时审慎考虑多方面因素。本研究为Copilot作为软件开发中AI结对程序员的作用提供了基于实证的基础和依据。