Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software development and replace human software developers. Nevertheless, there are in a lack of serious investigation into the capability of these LLM techniques in fulfilling software development tasks. In a controlled 2 $\times$ 2 between-subject experiment with 109 participants, we examined whether and to what degree working with ChatGPT was helpful in the coding task and typical software development task and how people work with ChatGPT. We found that while ChatGPT performed well in solving simple coding problems, its performance in supporting typical software development tasks was not that good. We also observed the interactions between participants and ChatGPT and found the relations between the interactions and the outcomes. Our study thus provides first-hand insights into using ChatGPT to fulfill software engineering tasks with real-world developers and motivates the need for novel interaction mechanisms that help developers effectively work with large language models to achieve desired outcomes.

翻译：近期，基于大型语言模型（LLM）的生成式AI在多个领域展现出令人印象深刻的高质量表现，尤其是在ChatGPT发布之后，其发展势头日益强劲。许多人认为，这些技术有潜力在软件开发中执行通用问题解决任务，并取代人类软件开发者。然而，目前缺乏对LLM技术完成软件开发任务能力的严谨研究。通过一项有109名参与者参与的受控2×2被试间实验，我们探究了使用ChatGPT在编码任务和典型软件开发任务中是否有帮助、帮助程度如何，以及人们如何与ChatGPT协作。研究发现，虽然ChatGPT在解决简单编码问题上表现良好，但在支持典型软件开发任务方面效果不佳。我们还观察了参与者与ChatGPT的交互过程，并发现了交互与结果之间的关系。因此，本研究提供了关于实际开发者使用ChatGPT完成软件工程任务的第一手见解，并揭示了开发新型交互机制的必要性，以帮助开发者有效利用大型语言模型实现预期目标。