The latest breakthroughs in large language models (LLM) have empowered software development tools, such as ChatGPT, to aid developers in complex tasks. Developers use ChatGPT to write code, review code changes, and even debug their programs. In these interactions, ChatGPT often recommends code snippets that depend on external libraries. However, code from libraries changes over time, invalidating a once-correct code snippet and making it difficult to reuse recommended code. In this study, we analyze DevGPT, a dataset of more than 4,000 Developer-ChatGPT interactions, to understand the role of library versions in code-related conversations. We quantify how often library version constraints are mentioned in code-related conversations and when ChatGPT recommends the installation of specific libraries. Our findings show that, albeit to constantly recommend and analyze code with external dependencies, library version constraints only appear in 9% of the conversations. In the majority of conversations, the version constraints are prompted by users (as opposed to being specified by ChatGPT) as a method for receiving better quality responses. Moreover, we study how library version constraints are used in the conversation through qualitative methods, identifying several potential problems that warrant further research.
翻译:大型语言模型(LLM)的最新突破赋能了诸如ChatGPT等软件开发工具,使其能够协助开发者完成复杂任务。开发者利用ChatGPT编写代码、审查代码更改,甚至调试程序。在这些交互中,ChatGPT常推荐依赖于外部库的代码片段。然而,库代码会随时间演变,这使得曾经正确的代码片段失效,并导致推荐代码难以复用。本研究通过分析包含4000余次开发者-ChatGPT交互的DevGPT数据集,探究库版本在代码相关对话中的作用。我们量化了代码相关对话中提及库版本约束的频率,以及ChatGPT何时推荐安装特定库。研究结果显示,尽管对话中频繁推荐和分析具有外部依赖的代码,但库版本约束仅出现在9%的对话中。在多数对话中,版本约束由用户主动提出(而非由ChatGPT指定),以此获取更高质量的反馈。此外,我们通过定性方法研究了库版本约束在对话中的使用模式,识别出若干有待进一步研究的潜在问题。