With the recent advancement of Artificial Intelligence (AI) and Large Language Models (LLMs), AI-based code generation tools become a practical solution for software development. GitHub Copilot, the AI pair programmer, utilizes machine learning models trained on a large corpus of code snippets to generate code suggestions using natural language processing. Despite its popularity in software development, there is limited empirical evidence on the actual experiences of practitioners who work with Copilot. To this end, we conducted an empirical study to understand the problems that practitioners face when using Copilot, as well as their underlying causes and potential solutions. We collected data from 473 GitHub issues, 706 GitHub discussions, and 142 Stack Overflow posts. Our results reveal that (1) Operation Issue and Compatibility Issue are the most common problems faced by Copilot users, (2) Copilot Internal Error, Network Connection Error, and Editor/IDE Compatibility Issue are identified as the most frequent causes, and (3) Bug Fixed by Copilot, Modify Configuration/Setting, and Use Suitable Version are the predominant solutions. Based on the results, we discuss the potential areas of Copilot for enhancement, and provide the implications for the Copilot users, the Copilot team, and researchers.
翻译:随着人工智能(AI)与大型语言模型(LLMs)的最新进展,基于AI的代码生成工具已成为软件开发的实用解决方案。AI结对编程工具GitHub Copilot利用在大量代码片段语料库上训练的机器学习模型,通过自然语言处理生成代码建议。尽管其在软件开发中广受欢迎,但关于开发者实际使用Copilot体验的实证证据仍然有限。为此,我们开展了一项实证研究,旨在理解开发人员在使用Copilot时面临的问题、其根本原因及潜在解决方案。我们收集了473个GitHub议题、706个GitHub讨论帖和142个Stack Overflow帖子的数据。研究结果表明:(1)操作问题与兼容性问题是Copilot用户最常遇到的问题;(2)Copilot内部错误、网络连接错误以及编辑器/IDE兼容性问题被确认为最频繁的成因;(3)由Copilot修复错误、修改配置/设置以及使用合适版本是主要的解决方案。基于这些结果,我们讨论了Copilot可改进的潜在方向,并为Copilot用户、Copilot开发团队及研究人员提供了启示。