Despite the advancements of open-source large language models (LLMs) and their variants, e.g., LLaMA and Vicuna, they remain significantly limited in performing higher-level tasks, such as following human instructions to use external tools (APIs). This is because current instruction tuning largely focuses on basic language tasks instead of the tool-use domain. This is in contrast to state-of-the-art (SOTA) LLMs, e.g., ChatGPT, which have demonstrated excellent tool-use capabilities but are unfortunately closed source. To facilitate tool-use capabilities within open-source LLMs, we introduce ToolLLM, a general tool-use framework of data construction, model training and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is created automatically using ChatGPT. Specifically, we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub, then prompt ChatGPT to generate diverse human instructions involving these APIs, covering both single-tool and multi-tool scenarios. Finally, we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To make the searching process more efficient, we develop a novel depth-first search-based decision tree (DFSDT), enabling LLMs to evaluate multiple reasoning traces and expand the search space. We show that DFSDT significantly enhances the planning and reasoning capabilities of LLMs. For efficient tool-use assessment, we develop an automatic evaluator: ToolEval. We fine-tune LLaMA on ToolBench and obtain ToolLLaMA. Our ToolEval reveals that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. To make the pipeline more practical, we devise a neural API retriever to recommend appropriate APIs for each instruction, negating the need for manual API selection.

翻译：尽管开源大型语言模型（LLMs）及其变体（如LLaMA和Vicuna）取得了进展，它们在执行高级任务（例如遵循人类指令使用外部工具/API）方面仍存在显著局限。这是因为当前的指令微调主要聚焦于基础语言任务，而非工具使用领域。这与最先进的（SOTA）LLMs（如ChatGPT）形成对比，后者展现出卓越的工具使用能力，但遗憾的是它们属于闭源模型。为提升开源LLMs的工具使用能力，我们提出了ToolLLM——一个涵盖数据构建、模型训练与评估的通用工具使用框架。我们首先介绍ToolBench，一个通过ChatGPT自动生成的工具使用指令微调数据集。具体而言，我们从RapidAPI Hub收集了涵盖49个类别的16,464个真实RESTful API，随后提示ChatGPT生成涉及这些API的多样化人类指令，覆盖单工具和多工具场景。最后，我们利用ChatGPT为每条指令搜索有效的解决方案路径（API调用链）。为提升搜索效率，我们开发了一种基于深度优先搜索的决策树（DFSDT），使LLMs能够评估多条推理路径并扩展搜索空间。实验表明，DFSDT显著增强了LLMs的规划与推理能力。为高效评估工具使用性能，我们设计了自动评估器ToolEval。我们在ToolBench上微调LLaMA，得到ToolLLaMA。ToolEval评估显示，ToolLLaMA在执行复杂指令、泛化至未见过的API方面展现出卓越能力，性能与ChatGPT相当。为使框架更具实用性，我们设计了神经API检索器，可为每条指令自动推荐合适的API，无需人工选择。