Applying large language models (LLMs) for academic API usage shows promise in reducing researchers' academic information seeking efforts. However, current LLM API-using methods struggle with complex API coupling commonly encountered in academic queries. To address this, we introduce SoAy, a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. The addition of the solution reduces the difficulty for the model to understand the complex relationships between APIs. Code improves the efficiency of reasoning. To evaluate SoAy, we introduce SoAyBench, an evaluation benchmark accompanied by SoAyEval, built upon a cloned environment of APIs from AMiner. Experimental results demonstrate a 34.58-75.99\% performance improvement compared to state-of-the-art LLM API-based baselines. All datasets, codes, tuned models, and deployed online services are publicly accessible at https://github.com/RUCKBReasoning/SoAy.
翻译:将大型语言模型应用于学术API使用,有望减少研究者在学术信息检索中的工作量。然而,当前的LLM API使用方法难以应对学术查询中常见的复杂API耦合问题。为此,我们提出了SoAy,一种面向学术信息检索的、基于解决方案的LLM API使用方法论。该方法采用带有解决方案的代码作为推理方式,其中解决方案是预先构建的API调用序列。解决方案的引入降低了模型理解API间复杂关系的难度,而代码则提升了推理效率。为评估SoAy,我们提出了SoAyBench评估基准及其配套工具SoAyEval,该基准建立在从AMiner复现的API环境之上。实验结果表明,相较于最先进的基于LLM API的基线方法,SoAy实现了34.58%至75.99%的性能提升。所有数据集、代码、调优模型及已部署的在线服务均公开于https://github.com/RUCKBReasoning/SoAy。