Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics. In this paper, we propose ToRA a series of Tool-integrated Reasoning Agents designed to solve challenging mathematical problems by seamlessly integrating natural language reasoning with the utilization of external tools (e.g., computation libraries and symbolic solvers), thereby amalgamating the analytical prowess of language and the computational efficiency of tools. To train ToRA, we curate interactive tool-use trajectories on mathematical datasets, apply imitation learning on the annotations, and propose output space shaping to further refine models' reasoning behavior. As a result, ToRA models significantly outperform open-source models on 10 mathematical reasoning datasets across all scales with 13%-19% absolute improvements on average. Notably, ToRA-7B reaches 44.6% on the competition-level dataset MATH, surpassing the best open-source model WizardMath-70B by 22% absolute. ToRA-34B is also the first open-source model that achieves an accuracy exceeding 50% on MATH, which significantly outperforms GPT-4's CoT result, and is competitive with GPT-4 solving problems with programs. Additionally, we conduct a comprehensive analysis of the benefits and remaining challenges of tool interaction for mathematical reasoning, providing valuable insights for future research.

翻译：大语言模型在各类语言任务中取得了显著进展，但在复杂数学问题上仍存在困难。本文提出ToRA（Tool-integrated Reasoning Agents）系列工具集成推理智能体，通过将自然语言推理与外部工具（如计算库和符号求解器）无缝结合来求解具有挑战性的数学问题，从而融合语言的分析能力与工具的计算效率。为训练ToRA，我们在数学数据集上构建了交互式工具使用轨迹，对标注数据应用模仿学习，并提出输出空间整形方法以进一步优化模型的推理行为。实验表明，ToRA模型在10个数学推理数据集上全面超越开源模型，在各规模层级上均实现13%-19%的绝对平均提升。特别地，ToRA-7B在竞赛级数据集MATH上达到44.6%的准确率，较最优开源模型WizardMath-70B提升22%的绝对值；ToRA-34B是首个在MATH上准确率突破50%的开源模型，显著超越GPT-4的思维链推理结果，并与GPT-4结合程序求解问题的方式具有竞争力。此外，本文对工具交互在数学推理中的优势与现存挑战进行了全面分析，为未来研究提供了重要启示。