Large Language Models (LLMs) leverage external tools primarily through generating the API request to enhance task completion efficiency. The accuracy of API request generation significantly determines the capability of LLMs to accomplish tasks. Due to the inherent hallucinations within the LLM, it is difficult to efficiently and accurately generate the correct API request. Current research uses prompt-based feedback to facilitate the LLM-based API request generation. However, existing methods lack factual information and are insufficiently detailed. To address these issues, we propose AutoFeedback, an LLM-based framework for efficient and accurate API request generation, with a Static Scanning Component (SSC) and a Dynamic Analysis Component (DAC). SSC incorporates errors detected in the API requests as pseudo-facts into the feedback, enriching the factual information. DAC retrieves information from API documentation, enhancing the level of detail in feedback. Based on this two components, Autofeedback implementes two feedback loops during the process of generating API requests by the LLM. Extensive experiments demonstrate that it significantly improves accuracy of API request generation and reduces the interaction cost. AutoFeedback achieves an accuracy of 100.00\% on a real-world API dataset and reduces the cost of interaction with GPT-3.5 Turbo by 23.44\%, and GPT-4 Turbo by 11.85\%.
翻译:大型语言模型(LLM)主要通过生成API请求来利用外部工具,以提升任务完成效率。API请求生成的准确性在很大程度上决定了LLM完成任务的能力。由于LLM固有的幻觉问题,高效且准确地生成正确的API请求较为困难。当前研究采用基于提示的反馈来促进基于LLM的API请求生成。然而,现有方法缺乏事实信息且细节不足。为解决这些问题,我们提出了AutoFeedback,这是一个基于LLM的高效准确API请求生成框架,包含静态扫描组件(SSC)和动态分析组件(DAC)。SSC将API请求中检测到的错误作为伪事实纳入反馈,以丰富事实信息。DAC从API文档中检索信息,增强反馈的细节水平。基于这两个组件,AutoFeedback在LLM生成API请求的过程中实现了两个反馈循环。大量实验表明,该方法显著提高了API请求生成的准确性并降低了交互成本。AutoFeedback在真实API数据集上达到了100.00%的准确率,并将与GPT-3.5 Turbo的交互成本降低了23.44%,与GPT-4 Turbo的交互成本降低了11.85%。