Recent advances in tool-integrated language agents have significantly improved their ability to solve complex reasoning tasks. However, existing alignment methods predominantly focus on maximizing task accuracy, while overlooking auxiliary objectives such as tool-use efficiency, which are essential for practical deployment. To address this gap, we introduce ParetoPO, a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives. In the first stage, ParetoPO leverages hypervolume-guided dynamic scalarization to adapt reward weights based on global Pareto frontier progress. In the second stage, it replaces scalarized learning signals with Pareto-ranking-based advantage computation, promoting nondominated trajectories through dominance-aware credit assignment. This design enables fine-grained, action-level optimization across multiple conflicting objectives. Experimental results on mathematic reasoning and multi-hop QA tasks show that ParetoPO consistently discovers policies with superior accuracy-efficiency trade-offs compared to static and heuristic baselines.

翻译：近期，集成了工具的语言智能体在解决复杂推理任务方面取得了显著进展。然而，现有对齐方法主要侧重于最大化任务准确性，而忽视了工具使用效率等对实际部署至关重要的辅助目标。为填补这一空白，我们提出了ParetoPO——一个用于对齐多目标竞争中工具性大语言模型（LLMs）的两阶段多目标优化框架。在第一阶段，ParetoPO利用超体积引导的动态标量化方法，根据全局帕累托前沿进展自适应调整奖励权重。在第二阶段，它用基于帕累托排序的优势计算替代标量化的学习信号，通过支配感知的信用分配促进非支配轨迹。该设计能够在多个相互冲突的目标上实现细粒度的、动作层面的优化。在数学推理和多跳问答任务上的实验结果表明，与静态和启发式基线方法相比，ParetoPO始终能够发现具有更优准确率-效率权衡的策略。