We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx

翻译：我们提出对话式网页导航问题，即数字代理通过控制网络浏览器，以多轮对话形式遵循用户指令解决真实世界任务。为支持该研究问题，我们创建了WEBLINX——一个包含2300次专家示范、涵盖10万次交互的大规模对话式网页导航基准测试。该基准覆盖150余个真实网站上的广泛模式，可应用于多样化场景下的代理训练与评估。由于网页信息量巨大，大型语言模型（LLMs）无法实时处理完整页面。为解决此瓶颈，我们设计了一种检索启发式模型，通过排序相关元素高效精简HTML页面。我们利用选定元素、屏幕截图及操作历史，评估多种模型在网页导航中复现人类行为的能力。实验涵盖从纯文本小模型到专有多模态大语言模型。研究发现：经过微调的小型解码器不仅超越最优零样本LLMs（包括GPT-4V），也优于显式预训练于屏幕截图的更大规模多模态微调模型。然而，所有微调模型均难以泛化至未见网站。本研究成果凸显了需开发能泛化至新场景的大规模多模态模型。我们的代码、数据与模型已开放供研究使用：https://mcgill-nlp.github.io/weblinx