Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.

翻译：在模拟环境中进行强化学习（RL）为提升基于语言的智能体提供了一种经济高效且高度可扩展的途径。然而，先前的研究局限于半自动化的环境合成或缺乏足够难度的任务，在广度与深度上均显不足。此外，集成于这些环境中的模拟用户的不稳定性，以及模拟环境之间的异质性，为智能体强化学习带来了进一步的挑战。在本工作中，我们提出：（1）一个用于自动化、可扩展地合成与高难度但易于验证任务相关联的模拟环境的统一流程；（2）一种环境层级的强化学习算法，该算法不仅能有效缓解用户不稳定性，还能在环境层级进行优势估计，从而提升训练效率与稳定性。在包括 tau-bench、tau2-Bench 和 VitaBench 在内的智能体基准测试上的全面评估验证了我们所提方法的有效性。进一步的深入分析凸显了其出色的领域外泛化能力。