Data science and engineering workflows often span multiple stages, from warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As vision language models (VLMs) advance in multimodal understanding and code generation, VLM-based agents could potentially automate these workflows by generating SQL queries, Python code, and GUI operations. This automation can improve the productivity of experts while democratizing access to large-scale data analysis. In this paper, we introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks, derived from real-world use cases, evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems. To balance realistic simulation with evaluation simplicity, we devote significant effort to developing automatic configurations for task setup and carefully crafting evaluation metrics for each task. Furthermore, we supplement multimodal agents with comprehensive documents of these enterprise data software systems. Our empirical evaluation reveals that existing state-of-the-art LLM/VLM-based agents do not reliably automate full data workflows (14.0% success). Even with step-by-step guidance, these agents still underperform in tasks that require fine-grained, knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted workspaces (10.6%). We hope that Spider2-V paves the way for autonomous multimodal agents to transform the automation of data science and engineering workflow. Our code and data are available at https://spider2-v.github.io.

翻译：数据科学与工程工作流通常涵盖从数据仓储到任务编排的多个阶段，涉及BigQuery、dbt、Airbyte等工具。随着视觉语言模型在多模态理解和代码生成方面的进步，基于视觉语言模型的智能体有望通过生成SQL查询、Python代码和图形界面操作来自动化这些工作流。这种自动化既能提升专家的工作效率，又能降低大规模数据分析的技术门槛。本文提出Spider2-V——首个专注于专业数据科学与工程工作流的多模态智能体基准测试，包含真实计算机环境中的494项实际任务，并整合了20个企业级专业应用程序。这些源自真实场景的任务，通过评估多模态智能体在企业数据软件系统中编写代码和管理图形界面的能力，来衡量其执行数据相关任务的综合水平。为在真实模拟与评估简便性之间取得平衡，我们投入大量精力开发任务自动配置方案，并为每类任务精心设计评估指标。此外，我们为多模态智能体提供了完整的企业数据软件系统文档支持。实证研究表明，现有基于大语言模型/视觉语言模型的先进智能体尚无法可靠地自动化完整数据工作流（成功率14.0%）。即使在逐步引导下，这些智能体在需要细粒度、知识密集型图形界面操作的任务（16.2%）以及涉及远程云端工作空间的任务（10.6%）中仍表现欠佳。我们希望Spider2-V能为自主多模态智能体革新数据科学与工程工作流自动化开辟道路。代码与数据已发布于https://spider2-v.github.io。