Large Language Models (LLMs) have shown promise in multiple software engineering tasks including code generation, program repair, code summarisation, and test generation. Fault localisation is instrumental in enabling automated debugging and repair of programs and was prominently featured as a highlight during the launch event of ChatGPT-4. Nevertheless, the performance of LLMs compared to state-of-the-art methods, as well as the impact of prompt design and context length on their efficacy, remains unclear. To fill this gap, this paper presents an in-depth investigation into the capability of ChatGPT-3.5 and ChatGPT-4, the two state-of-the-art LLMs, on fault localisation. Using the widely-adopted large-scale Defects4J dataset, we compare the two LLMs with the existing fault localisation techniques. We also investigate the consistency of LLMs in fault localisation, as well as how prompt engineering and the length of code context affect the fault localisation effectiveness. Our findings demonstrate that within function-level context, ChatGPT-4 outperforms all the existing fault localisation methods. Additional error logs can further improve ChatGPT models' localisation accuracy and consistency, with an average 46.9% higher accuracy over the state-of-the-art baseline SmartFL on the Defects4J dataset in terms of TOP-1 metric. However, when the code context of the Defects4J dataset expands to the class-level, ChatGPT-4's performance suffers a significant drop, with 49.9% lower accuracy than SmartFL under TOP-1 metric. These observations indicate that although ChatGPT can effectively localise faults under specific conditions, limitations are evident. Further research is needed to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.

翻译：大型语言模型（LLMs）在代码生成、程序修复、代码摘要和测试生成等多个软件工程任务中展现出潜力。故障定位作为自动化调试和程序修复的关键技术，在ChatGPT-4的发布活动中被重点强调。然而，LLMs相较于现有最优方法的性能表现，以及提示设计和上下文长度对其效能的影响仍不明确。为填补这一研究空白，本文深入探究了ChatGPT-3.5和ChatGPT-4这两个最新LLMs在故障定位任务中的能力。通过采用广泛使用的大规模Defects4J数据集，我们将这两个LLMs与现有故障定位技术进行对比，同时考察了LLMs在故障定位中的一致性，以及提示工程设计和代码上下文长度对故障定位效果的影响。研究结果表明，在函数级上下文中，ChatGPT-4优于所有现有故障定位方法。附加的错误日志能进一步提升ChatGPT模型的定位准确性和一致性，在Defects4J数据集上以TOP-1指标衡量，其准确率较当前最优基准SmartFL平均提高46.9%。然而，当Defects4J数据集的代码上下文扩展至类级别时，ChatGPT-4的性能显著下降，在TOP-1指标下准确率比SmartFL低49.9%。这些观察结果表明，尽管ChatGPT在特定条件下能有效定位故障，但其局限性也显而易见。要充分发挥ChatGPT等LLMs在实际故障定位应用中的潜力，仍需进一步研究。