Programmers often seek help from Q\&A websites to resolve issues they encounter during programming. Stack Overflow has been a widely used platform for this purpose for over a decade. Recently, revolutionary AI-powered platforms like ChatGPT have quickly gained popularity among programmers for their efficient and personalized programming assistance via natural language interactions. Both platforms can offer valuable assistance to programmers, but it's unclear which is more effective at enhancing programmer productivity. In our paper, we conducted an exploratory user study to compare the performance of Stack Overflow and ChatGPT in enhancing programmer productivity. Two groups of students with similar programming abilities were instructed to use the two platforms to solve three different types of programming tasks: algorithmic challenges, library usage, and debugging. During the experiments, we measured and compared the quality of code produced and the time taken to complete tasks for the two groups. The results show that, concerning code quality, ChatGPT outperforms Stack Overflow significantly in helping complete algorithmic and library-related tasks, while Stack Overflow is better for debugging tasks. Regarding task completion speed, the ChatGPT group is obviously faster than the Stack Overflow group in the algorithmic challenge, but the two groups have a similar performance in the other two tasks. Additionally, we conducted a post-experiment survey with the participants to understand how the platforms have helped them complete the programming tasks. We analyzed the questionnaires to summarize ChatGPT and Stack Overflow's strengths and weaknesses pointed out by the participants. By comparing these, we identified the reasons behind the two platforms' divergent performances in programming assistance.
翻译:程序员在编程过程中常通过问答网站寻求帮助以解决遇到的问题。Stack Overflow作为此类平台已广泛使用超过十年。近年来,ChatGPT等革命性人工智能平台凭借其通过自然语言交互提供高效个性化编程辅助的能力,迅速受到程序员青睐。这两个平台都能为程序员提供宝贵帮助,但尚不明确哪个平台在提升程序员生产力方面更为有效。本文通过探索性用户研究,对比了Stack Overflow与ChatGPT在提升程序员生产力方面的表现。我们将编程能力相近的两组学生分别使用这两个平台完成三类不同编程任务:算法挑战、库函数使用及调试。实验过程中,我们测量并比较了两组学生产出代码的质量与任务完成时间。结果显示,在代码质量方面,ChatGPT在协助完成算法类和库函数任务时显著优于Stack Overflow,而Stack Overflow在调试任务中表现更佳;在任务完成速度方面,ChatGPT组在算法挑战任务中明显快于Stack Overflow组,但两组在其他两类任务中表现相近。此外,我们开展了实验后问卷调查,了解平台如何帮助参与者完成编程任务。通过分析问卷,我们总结了参与者指出的ChatGPT与Stack Overflow的优势与不足,并通过对比揭示了两平台在编程辅助方面表现差异的原因。