With the recent advancement of Artificial Intelligence (AI) and the emergence of Large Language Models (LLMs), AI-based code generation tools have achieved significant progress and become a practical solution for software development. GitHub Copilot, referred to as AI pair programmer, utilizes machine learning models that are trained on a large corpus of code snippets to generate code suggestions or auto-complete code using natural language processing. Despite its popularity, there is little empirical evidence on the actual experiences of software developers who work with Copilot. To this end, we conducted an empirical study to understand the issues and challenges that developers face when using Copilot in practice, as well as their underlying causes and potential solutions. We collected data from 476 GitHub issues, 706 GitHub discussions, and 184 Stack Overflow posts, and identified the issues, causes that trigger the issues, and solutions that resolve the issues when using Copilot. Our results reveal that (1) Usage Issue and Compatibility Issue are the most common problems faced by Copilot users, (2) Copilot Internal Issue, Network Connection Issue, and Editor/IDE Compatibility Issue are identified as the most frequent causes, and (3) Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version are the predominant solutions. Based on the results, we delve into the main challenges users encounter when implementing Copilot in practical development, the possible impact of Copilot on the coding process, aspects in which Copilot can be further enhanced, and potential new features desired by Copilot users.
翻译:随着人工智能的近期进展及大型语言模型的出现,基于AI的代码生成工具已取得显著进步,并成为软件开发的实用解决方案。GitHub Copilot(被称为AI结对程序员)利用在大规模代码片段语料库上训练的机器学习模型,通过自然语言处理生成代码建议或自动补全代码。尽管Copilot广受欢迎,但关于软件开发者实际使用体验的实证证据仍较为匮乏。为此,我们开展了一项实证研究,旨在理解开发者在实践中使用Copilot时遇到的问题与挑战、根本原因及潜在解决方案。我们从476个GitHub议题、706个GitHub讨论帖和184个Stack Overflow帖子中采集数据,识别出Copilot使用中的问题、触发原因及解决方案。研究结果表明:(1)使用问题与兼容性问题是最常见的两类问题;(2)Copilot内部问题、网络连接问题及编辑器/IDE兼容性问题是最频繁出现的成因;(3)Copilot自身修复的漏洞、修改配置/设置及使用合适版本是主要的解决方案。基于研究结果,我们深入探讨了用户在实际开发中部署Copilot时面临的主要挑战、Copilot对编码流程的潜在影响、可进一步优化的环节,以及用户期望的新功能特性。