Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs' question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs' internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs' ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs and suggest new directions for future advancements. Our data and code are freely available to the broader scientific community on GitHub.

翻译：大语言模型（LLMs）在各类自然语言处理任务中展现出卓越性能，但仍面临幻觉现象与数值推理能力薄弱等挑战。为克服这些难题，可借助外部工具增强LLMs的问答能力。然而，现有评估方法未能有效区分基于LLMs内部知识可回答的问题与需通过工具调用外部信息才能解答的问题。针对该问题，我们提出名为ToolQA的新数据集，旨在忠实评估LLMs运用外部工具进行问答的能力。ToolQA的开发包含可扩展的自动化数据集构建流程，并配套13种专用工具实现与外部知识的交互以解答问题。值得强调的是，我们力求最小化基准数据与LLMs预训练数据之间的重叠，从而更精准地评估LLMs的工具推理能力。通过深度诊断现有工具型LLMs，我们揭示了其优势、不足及潜在改进方向。本研究为LLMs评估树立了新基准，并为未来发展指明新路径。相关数据与代码已在GitHub上向科学界免费开放。