Large Language Models (LLMs) have shown promise in multiple software engineering tasks including code generation, code summarisation, test generation and code repair. Fault localisation is essential for facilitating automatic program debugging and repair, and is demonstrated as a highlight at ChatGPT-4's launch event. Nevertheless, there has been little work understanding LLMs' capabilities for fault localisation in large-scale open-source programs. To fill this gap, this paper presents an in-depth investigation into the capability of ChatGPT-3.5 and ChatGPT-4, the two state-of-the-art LLMs, on fault localisation. Using the widely-adopted Defects4J dataset, we compare the two LLMs with the existing fault localisation techniques. We also investigate the stability and explanation of LLMs in fault localisation, as well as how prompt engineering and the length of code context affect the fault localisation effectiveness. Our findings demonstrate that within a limited code context, ChatGPT-4 outperforms all the existing fault localisation methods. Additional error logs can further improve ChatGPT models' localisation accuracy and stability, with an average 46.9% higher accuracy over the state-of-the-art baseline SmartFL in terms of TOP-1 metric. However, performance declines dramatically when the code context expands to the class-level, with ChatGPT models' effectiveness becoming inferior to the existing methods overall. Additionally, we observe that ChatGPT's explainability is unsatisfactory, with an accuracy rate of only approximately 30%. These observations demonstrate that while ChatGPT can achieve effective fault localisation performance under certain conditions, evident limitations exist. Further research is imperative to fully harness the potential of LLMs like ChatGPT for practical fault localisation applications.

翻译：大型语言模型（LLMs）在多项软件工程任务中展现出潜力，包括代码生成、代码摘要、测试生成和代码修复。故障定位是促进自动化程序调试与修复的关键环节，在ChatGPT-4的发布活动中被作为亮点展示。然而，目前关于LLMs在大型开源程序中故障定位能力的研究尚属匮乏。为弥补这一空白，本文针对两种最先进的LLMs——ChatGPT-3.5和ChatGPT-4——在故障定位上的能力进行了深入探究。利用广泛采用的Defects4J数据集，我们将这两种LLMs与现有故障定位技术进行了比较。同时，我们研究了LLMs在故障定位中的稳定性与可解释性，以及提示工程和代码上下文长度对故障定位效果的影响。研究结果表明，在有限的代码上下文范围内，ChatGPT-4的性能优于所有现有故障定位方法。附加的错误日志可进一步提升ChatGPT模型的定位准确性与稳定性，在TOP-1指标上平均比最先进的基线方法SmartFL高出46.9%。然而，当代码上下文扩展至类级别时，性能显著下降，ChatGPT模型的整体效果劣于现有方法。此外，我们观察到ChatGPT的可解释性不尽如人意，准确率仅为约30%。这些观察表明，尽管ChatGPT在特定条件下能实现有效的故障定位性能，但仍存在明显局限。未来亟需进一步研究，以充分挖掘ChatGPT等LLMs在实际故障定位应用中的潜力。