ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot launched by OpenAI on November 30, 2022. OpenAI's GPT-3 family of large language models serve as the foundation for ChatGPT. ChatGPT is fine-tuned with both supervised and reinforcement learning techniques and has received widespread attention for its articulate responses across diverse domains of knowledge. In this study, we explore how ChatGPT can be used to help with common software engineering tasks. Many of the ubiquitous tasks covering the breadth of software engineering such as ambiguity resolution in software requirements, method name suggestion, test case prioritization, code review, log summarization can potentially be performed using ChatGPT. In this study, we explore fifteen common software engineering tasks using ChatGPT. We juxtapose and analyze ChatGPT's answers with the respective state of the art outputs (where available) and/or human expert ground truth. Our experiments suggest that for many tasks, ChatGPT does perform credibly and the response from it is detailed and often better than the human expert output or the state of the art output. However, for a few other tasks, ChatGPT in its present form provides incorrect answers and hence is not suited for such tasks.
翻译:ChatGPT(生成式预训练变换器聊天机器人)是由OpenAI于2022年11月30日发布的聊天机器人。OpenAI的GPT-3系列大型语言模型构成了ChatGPT的基础。ChatGPT通过监督学习和强化学习技术进行微调,并因其在多元知识领域的清晰回答而受到广泛关注。在本研究中,我们探讨了如何利用ChatGPT协助完成常见的软件工程任务。许多覆盖软件工程全范围的泛在任务,例如软件需求中的歧义消解、方法名建议、测试用例优先级排序、代码审查、日志摘要等,都可能借助ChatGPT实现。本研究利用ChatGPT探索了十五项常见软件工程任务。我们将ChatGPT的答案与各自领域的最优输出(如可用)和/或人类专家基准进行对比和分析。实验表明,对于许多任务,ChatGPT确实表现可靠,其回答详细且通常优于人类专家输出或最优输出。然而,对于其他少数任务,ChatGPT在当前形式下提供了错误答案,因此不适用于此类任务。