Software repositories contain valuable information for gaining insights into their development process. However, extracting insights from these repository data is time-consuming and requires technical expertise. While software engineering chatbots have been developed to facilitate natural language interactions with repositories, they struggle with understanding natural language and accurately retrieving relevant data. This study aims to improve the accuracy of LLM-based chatbots in answering repository-related questions by augmenting them with knowledge graphs. We achieve this in a two-step approach; (1) constructing a knowledge graph from the repository data and (2) synergizing the knowledge graph with LLM to allow for the natural language questions and answers. We curated a set of 20 questions with different complexities and evaluated our approach on five popular open-source projects. Our approach achieved an accuracy of 65%. We further investigated the limitations and identified six key issues, with the majority relating to the reasoning capability of the LLM. We experimented with a few-shot chain-of-thought prompting to determine if it could enhance our approach. This technique improved the overall accuracy to 84%. Our findings demonstrate the synergy between LLMs and knowledge graphs as a viable solution for making repository data accessible to both technical and non-technical stakeholders.

翻译：软件仓库蕴含了洞察其开发过程的宝贵信息。然而，从这些仓库数据中提取洞见既耗时又需要专业技术。尽管已有软件工程聊天机器人被开发出来以促进与仓库的自然语言交互，但它们在理解自然语言和准确检索相关数据方面仍存在困难。本研究旨在通过用知识图谱增强大语言模型，提高基于大语言模型的聊天机器人在回答仓库相关问题时的准确性。我们通过两步法实现这一目标：(1) 从仓库数据构建知识图谱；(2) 将知识图谱与大语言模型协同，以实现自然语言问答。我们整理了一套包含20个不同复杂度的问题集，并在五个流行的开源项目上评估了我们的方法。我们的方法达到了65%的准确率。我们进一步探究了其局限性，并识别出六个关键问题，其中大部分与大语言模型的推理能力有关。我们尝试使用少样本思维链提示技术，以确定其是否能提升我们的方法。该技术将整体准确率提升至84%。我们的研究结果表明，大语言模型与知识图谱之间的协同作用，是使技术及非技术相关方都能访问仓库数据的一种可行解决方案。