Graphical User Interface (GUI) agents are designed to automate complex tasks on digital devices, such as smartphones and desktops. Most existing GUI agents interact with the environment through extracted structured data, which can be notably lengthy (e.g., HTML) and occasionally inaccessible (e.g., on desktops). To alleviate this issue, we propose a visual GUI agent -- SeeClick, which only relies on screenshots for task automation. In our preliminary study, we have discovered a key challenge in developing visual GUI agents: GUI grounding -- the capacity to accurately locate screen elements based on instructions. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. Along with the efforts above, we have also created ScreenSpot, the first realistic GUI grounding dataset that encompasses mobile, desktop, and web environments. After pre-training, SeeClick demonstrates significant improvement in ScreenSpot over various baselines. Moreover, comprehensive evaluations on three widely used benchmarks consistently support our finding that advancements in GUI grounding directly correlate with enhanced performance in downstream GUI agent tasks. The model, data and code are available at https://github.com/njucckevin/SeeClick.

翻译：图形用户界面（GUI）智能体旨在自动化智能手机、桌面电脑等数字设备上的复杂任务。现有大多GUI智能体通过提取结构化数据（如HTML）与环境交互，这类数据往往冗长且在某些场景（如桌面环境）中难以获取。为解决此问题，我们提出视觉型GUI智能体——SeeClick，该智能体仅依赖截图即可实现任务自动化。初步研究中，我们发现开发视觉GUI智能体的关键挑战在于：GUI定位能力——即根据指令精准定位屏幕元素的能力。为此，我们提出通过GUI定位预训练增强SeeClick，并设计了一种自动化构建GUI定位数据的方法。基于上述努力，我们还创建了首个覆盖移动端、桌面端及网页环境的真实GUI定位数据集ScreenSpot。经预训练后，SeeClick在ScreenSpot上的表现显著优于多种基线方法。此外，在三个广泛使用的基准测试中的综合评估一致表明：GUI定位能力的提升与下游GUI智能体任务的性能增强存在直接关联。模型、数据及代码已开源至https://github.com/njucckevin/SeeClick。