Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.

翻译：基于大语言模型的工具使用智能体在数学推理和多跳问答等任务中表现出色。然而，在长轨迹场景中，智能体常会触发过多且低质量的工具调用，导致延迟增加和推理性能下降，使得工具使用行为的管理变得具有挑战性。本文通过基于熵的探索性实验，观察到熵降低与高质量工具调用之间存在强正相关关系。基于这一发现，我们提出将熵降低作为监督信号，并设计了两种奖励策略以优化工具使用行为的不同需求：稀疏结果奖励提供粗略的轨迹级指导以提高效率，而密集过程奖励则提供细粒度的监督以提升性能。跨多个领域的实验表明，两种奖励设计均能改善工具使用行为：前者相比基线平均减少72.07%的工具调用量，后者将性能提升22.27%。这些结果将熵降低定位为增强工具使用行为的关键机制，使智能体在现实应用中更具适应性。