Graphical User Interface (GUI) automation holds significant promise for enhancing human productivity by assisting with computer tasks. Existing task formulations primarily focus on simple tasks that can be specified by a single, language-only instruction, such as "Insert a new slide." In this work, we introduce VideoGUI, a novel multi-modal benchmark designed to evaluate GUI assistants on visual-centric GUI tasks. Sourced from high-quality web instructional videos, our benchmark focuses on tasks involving professional and novel software (e.g., Adobe Photoshop or Stable Diffusion WebUI) and complex activities (e.g., video editing). VideoGUI evaluates GUI assistants through a hierarchical process, allowing for identification of the specific levels at which they may fail: (i) high-level planning: reconstruct procedural subtasks from visual conditions without language descriptions; (ii) middle-level planning: generate sequences of precise action narrations based on visual state (i.e., screenshot) and goals; (iii) atomic action execution: perform specific actions such as accurately clicking designated elements. For each level, we design evaluation metrics across individual dimensions to provide clear signals, such as individual performance in clicking, dragging, typing, and scrolling for atomic action execution. Our evaluation on VideoGUI reveals that even the SoTA large multimodal model GPT4o performs poorly on visual-centric GUI tasks, especially for high-level planning.

翻译：图形用户界面（GUI）自动化通过协助完成计算机任务，在提升人类生产力方面具有重要潜力。现有的任务形式主要集中于可通过单一纯语言指令指定的简单任务，例如“插入新幻灯片”。本研究提出了VideoGUI——一个新颖的多模态基准测试，旨在评估GUI助手在以视觉为中心的GUI任务上的表现。该基准测试源自高质量网络教学视频，重点关注涉及专业及新型软件（如Adobe Photoshop或Stable Diffusion WebUI）和复杂活动（如视频编辑）的任务。VideoGUI通过分层流程评估GUI助手，能够识别其可能失败的具体层级：（i）高层规划：在无语言描述的情况下，根据视觉条件重构程序性子任务；（ii）中层规划：基于视觉状态（即屏幕截图）与目标生成精确动作叙述序列；（iii）原子动作执行：执行特定操作，如准确点击指定元素。针对每个层级，我们设计了跨维度的评估指标以提供清晰信号，例如在原子动作执行层面分别评估点击、拖拽、输入和滚动的独立性能。我们在VideoGUI上的评估表明，即使是当前最先进的大型多模态模型GPT4o，在以视觉为中心的GUI任务上表现仍然欠佳，尤其是在高层规划方面。