The advancement of large language models (LLMs) leads to a new era marked by the development of autonomous applications in the real world, which drives innovation in the creation of advanced web-based agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we propose a new evaluation protocol for web agents to address the challenges of automatic evaluation of open-ended web agent tasks, leveraging the robust multimodal comprehension capabilities of GPT-4V. We create a new benchmark by gathering real-world tasks from 15 widely used websites to evaluate our agents. We show that WebVoyager achieves a 55.7% task success rate, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager in practical applications. We found that our proposed automatic evaluation achieves 85.3% agreement with human judgment, paving the way for further development of web agents in a real-world setting.

翻译：大语言模型的进步开启了真实世界自主应用发展的新时代，推动了先进Web智能体的创新研究。现有Web智能体通常仅处理单一输入模态，且仅在简化版Web模拟器或静态网页快照中进行评估，这极大限制了它们在真实场景中的适用性。为弥合这一差距，我们提出WebVoyager——一种基于大规模多模态模型（LMM）的创新Web智能体，能够通过与真实网站交互，端到端地完成用户指令。此外，我们为Web智能体设计了一种新的评估协议，利用GPT-4V强大的多模态理解能力，解决开放式Web智能体任务的自动评估挑战。通过收集来自15个广泛使用网站的真实任务，我们构建了新的基准以评估智能体性能。实验表明，WebVoyager取得了55.7%的任务成功率，显著超越GPT-4（完整工具集）及文本版WebVoyager配置，突显了该智能体在实际应用中的卓越能力。我们还发现，所提出的自动评估方法与人工判断的一致性达到85.3%，为Web智能体在真实场景中的进一步发展铺平了道路。