API recommendation methods have evolved from literal and semantic keyword matching to query expansion and query clarification. The latest query clarification method is knowledge graph (KG)-based, but limitations include out-of-vocabulary (OOV) failures and rigid question templates. To address these limitations, we propose a novel knowledge-guided query clarification approach for API recommendation that leverages a large language model (LLM) guided by KG. We utilize the LLM as a neural knowledge base to overcome OOV failures, generating fluent and appropriate clarification questions and options. We also leverage the structured API knowledge and entity relationships stored in the KG to filter out noise, and transfer the optimal clarification path from KG to the LLM, increasing the efficiency of the clarification process. Our approach is designed as an AI chain that consists of five steps, each handled by a separate LLM call, to improve accuracy, efficiency, and fluency for query clarification in API recommendation. We verify the usefulness of each unit in our AI chain, which all received high scores close to a perfect 5. When compared to the baselines, our approach shows a significant improvement in MRR, with a maximum increase of 63.9% higher when the query statement is covered in KG and 37.2% when it is not. Ablation experiments reveal that the guidance of knowledge in the KG and the knowledge-guided pathfinding strategy are crucial for our approach's performance, resulting in a 19.0% and 22.2% increase in MAP, respectively. Our approach demonstrates a way to bridge the gap between KG and LLM, effectively compensating for the strengths and weaknesses of both.
翻译:API推荐方法已从字面与语义关键词匹配演进至查询扩展与查询澄清。最新的查询澄清方法基于知识图谱,但存在词表外失效与僵化的问题模板等局限。针对这些局限,我们提出了一种新颖的知识引导型查询澄清方法,该方法利用由知识图谱引导的大语言模型进行API推荐。我们将大语言模型作为神经知识库以克服词表外失效,生成流畅且恰当的澄清问题与选项。同时,我们利用知识图谱中存储的结构化API知识与实体关系来过滤噪声,并将最优澄清路径从知识图谱迁移至大语言模型,从而提高澄清过程的效率。本方法设计为一个由五步组成的AI链,每一步由单独的大语言模型调用处理,以提升API推荐中查询澄清的准确性、效率与流畅性。我们验证了AI链中各单元的有效性,所有单元均获得接近满分的5分高评分。与基线方法相比,本方法在平均倒数排名(MRR)上呈现显著提升:当查询语句覆盖于知识图谱时最大提升63.9%,未覆盖时最大提升37.2%。消融实验显示,知识图谱中的知识引导以及知识引导的路径发现策略对本方法的性能至关重要,分别使平均精度均值(MAP)提升19.0%与22.2%。本方法展现了弥合知识图谱与大语言模型之间差距的路径,有效互补了二者的优势与局限。