Augmenting large language models (LLMs) with external tools is a promising approach to enhance their capabilities, especially for complex tasks. Synthesizing tool-use data through real-world simulations is an effective way to achieve this. However, our investigation reveals that training gains significantly decay as synthetic data increases. The model struggles to benefit from additional synthetic data, which fails to endow it with advanced tool-use capabilities in complex scenarios Moreover, we discovered that the above limitation usually manifests as a fragment deficiency (i.e., parameter errors) in response. To this end, we propose an iterative reinforced fine-tuning strategy designed to alleviate this limitation. This strategy involves: (1) enhancing the diversity of response for synthetic data through path exploration of Monte Carlo Tree Search. (2) iteratively pinpointing the model's deficiency by constructing fine-grained preference pairs, and then improving it by preference optimization algorithms for targeted improvement. The experiments show that our method achieves 13.11% better performance than the same-size base model. It achieves an improvement of 6.5% in complex scenarios compared to the baseline, and it also outperforms larger open-source and closed-source models.

翻译：通过外部工具增强大型语言模型（LLMs）是提升其能力、特别是应对复杂任务的一种有前景的方法。通过现实世界模拟合成工具使用数据是实现这一目标的有效途径。然而，我们的研究发现，随着合成数据的增加，训练收益显著衰减。模型难以从额外的合成数据中获益，这导致其无法在复杂场景中获得高级工具使用能力。此外，我们发现上述限制通常表现为响应中的片段缺陷（即参数错误）。为此，我们提出了一种迭代强化微调策略，旨在缓解这一限制。该策略包括：（1）通过蒙特卡洛树搜索的路径探索增强合成数据响应的多样性。（2）通过构建细粒度偏好对迭代定位模型的缺陷，并利用偏好优化算法进行针对性改进。实验表明，我们的方法比同规模基础模型性能提升13.11%，在复杂场景中较基线提升6.5%，并且优于更大的开源和闭源模型。