Evaluation remains a critical bottleneck for interactive agent development. Existing evaluation methods often rely on static benchmarks, which fail to capture the dynamic, multi-step nature of agentic behavior and struggle to expose meaningful failure modes. While user-simulation-based evaluation offers a promising alternative, existing simulation frameworks suffer from two major limitations. First, they provide limited mechanisms for evaluating the quality and comprehensiveness of simulated interactions, making it difficult to assess whether a simulator sufficiently explores an agent's capabilities and failure modes. Second, most frameworks are restricted to either UI-only actions or API-only actions, limiting their ability to model the full range of realistic user behaviors. To address these limitations, we propose VISTA, a Versatile Interactive user Simulation Toolkit for Agent evaluation. Our toolkit includes a suite of six metrics for measuring the realism, capability coverage, and interaction effectiveness of simulated interactions. In addition, we develop a hybrid user simulator that integrates both UI-based interactions and API-based interactions, enabling more realistic and comprehensive evaluation across diverse interactive environments. We evaluate VISTA in e-commerce shopping and education customer service settings and demonstrate that it produces more realistic and comprehensive evaluations than existing methods.

翻译：评估仍然是交互式智能体开发的关键瓶颈。现有评估方法通常依赖静态基准测试，难以捕捉智能体行为的动态多步性质，也难以揭示有意义的失败模式。尽管基于用户模拟的评估提供了一种有前景的替代方案，但现有模拟框架存在两大局限。首先，它们缺乏评估模拟交互质量与全面性的机制，难以判断模拟器是否充分探索了智能体的能力及失败模式。其次，大多数框架仅局限于纯界面（UI）操作或纯API操作，限制了其对真实用户行为全谱的建模能力。为应对这些局限，我们提出VISTA——面向智能体评估的通用交互式用户模拟工具包。该工具包包含六项用于衡量模拟交互真实性、能力覆盖率和交互有效性的指标。此外，我们开发了一种混合型用户模拟器，该模拟器融合了基于界面的交互与基于API的交互，从而能够在多样化的交互环境中实现更真实、更全面的评估。我们在电子商务购物和教育客服场景中评估了VISTA，结果表明其能产生比现有方法更真实、更全面的评估结果。