With generative AI advances, the exciting potential for autonomous agents to manage daily tasks via natural language commands has emerged. However, cur rent agents are primarily created and tested in simplified synthetic environments, substantially limiting real-world scenario representation. In this paper, we build an environment for agent command and control that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on websites, and we create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and are designed to emulate tasks that humans routinely perform on the internet. We design and implement several autonomous agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 10.59%. These results highlight the need for further development of robust agents, that current state-of-the-art LMs are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress. Our code, data, environment reproduction resources, and video demonstrations are publicly available at https://webarena.dev/.

翻译：随着生成式人工智能的进步，通过自然语言命令让自主智能体管理日常任务的激动人心的潜力已经显现。然而，当前的智能体主要是在简化的合成环境中创建和测试的，这极大地限制了真实世界场景的表示。在本文中，我们构建了一个高度真实且可复现的智能体指挥与控制环境。具体而言，我们专注于能执行网站任务的智能体，并创建了一个包含来自四个常见领域（电子商务、社交论坛讨论、协作软件开发和内容管理）的完整功能网站的环境。我们的环境配备了工具（例如地图）和外部知识库（例如用户手册），以鼓励类似人类的任务解决方式。基于此环境，我们发布了一组基准测试任务，侧重于评估任务完成的功能正确性。我们基准测试中的任务多样化、周期长，并旨在模拟人类在互联网上日常执行的任务。我们设计并实现了多个自主智能体，整合了诸如“先推理后行动”等最新技术。结果表明，解决复杂任务具有挑战性：我们基于GPT-4的最佳智能体仅实现了10.59%的端到端任务成功率。这些结果凸显了开发更鲁棒智能体的必要性，当前最先进的语言模型在这些现实任务中远未达到完美表现，而WebArena可用于衡量此类进展。我们的代码、数据、环境复现资源及视频演示已在https://webarena.dev/上公开提供。