The rapid advancement of large language models (LLMs) has led to a new era marked by the development of autonomous applications in real-world scenarios, which drives innovation in creating advanced web agents. Existing web agents typically only handle one input modality and are evaluated only in simplified web simulators or static web snapshots, greatly limiting their applicability in real-world scenarios. To bridge this gap, we introduce WebVoyager, an innovative Large Multimodal Model (LMM) powered web agent that can complete user instructions end-to-end by interacting with real-world websites. Moreover, we establish a new benchmark by compiling real-world tasks from 15 popular websites and introduce an automatic evaluation protocol leveraging multimodal understanding abilities of GPT-4V to evaluate open-ended web agents. We show that WebVoyager achieves a 59.1% task success rate on our benchmark, significantly surpassing the performance of both GPT-4 (All Tools) and the WebVoyager (text-only) setups, underscoring the exceptional capability of WebVoyager. The proposed automatic evaluation metric achieves 85.3% agreement with human judgment, indicating its effectiveness in providing reliable and accurate assessments of web agents.

翻译：大型语言模型的快速发展标志着自主应用在真实场景中的开发进入新时代，推动着先进网络智能体的创新。现有网络智能体通常仅处理单一输入模态，且仅在简化网络模拟器或静态网页快照中进行评估，极大限制了其在真实场景中的适用性。为弥补这一缺陷，我们提出WebVoyager——一种由大型多模态模型驱动的创新网络智能体，可通过与真实网站交互端到端地完成用户指令。此外，我们通过整合15个热门网站的真实任务建立新基准，并引入利用GPT-4V多模态理解能力的自动评估协议来评测开放型网络智能体。实验表明，WebVoyager在基准测试中达到59.1%的任务成功率，显著超越GPT-4（全工具版）及纯文本版WebVoyager的性能，突显其卓越能力。所提出的自动评估指标与人工判断的一致性达85.3%，验证了其为网络智能体提供可靠准确评估的有效性。