Digital agents are increasingly employed to automate tasks in interactive digital environments such as web pages, software applications, and operating systems. While text-based agents built on Large Language Models (LLMs) often require frequent updates due to platform-specific APIs, visual agents leveraging Multimodal Large Language Models (MLLMs) offer enhanced adaptability by interacting directly with Graphical User Interfaces (GUIs). However, these agents face significant challenges in visual perception, particularly when handling high-resolution, visually complex digital environments. This paper introduces Iris, a foundational visual agent that addresses these challenges through two key innovations: Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL). ISC dynamically identifies and prioritizes visually dense regions using a edge detection algorithm, enabling efficient processing by allocating more computational resources to areas with higher information density. SRDL enhances the agent's ability to handle complex tasks by leveraging a dual-learning loop, where improvements in referring (describing UI elements) reinforce grounding (locating elements) and vice versa, all without requiring additional annotated data. Empirical evaluations demonstrate that Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations, outperforming methods using 10x more training data. These improvements further translate to significant gains in both web and OS agent downstream tasks.

翻译：数字代理正日益广泛地应用于自动化交互式数字环境（如网页、软件应用和操作系统）中的任务。尽管基于大型语言模型（LLMs）的文本代理常因平台专用API而需要频繁更新，但利用多模态大型语言模型（MLLMs）的视觉代理通过直接与图形用户界面（GUIs）交互，提供了更强的适应性。然而，这些代理在视觉感知方面面临重大挑战，尤其是在处理高分辨率、视觉复杂度高的数字环境时。本文提出Iris——一种基础视觉代理，通过两项关键创新应对这些挑战：信息敏感裁剪（ISC）与自我精炼双重学习（SRDL）。ISC利用边缘检测算法动态识别并优先处理视觉密集区域，通过向信息密度更高的区域分配更多计算资源实现高效处理。SRDL通过双重学习循环增强代理处理复杂任务的能力，其中指代表述（描述UI元素）的改进会强化定位（确定元素位置）能力，反之亦然，且整个过程无需额外标注数据。实证评估表明，Iris仅使用85万GUI标注就在多个基准测试中实现了最先进的性能，其表现优于使用10倍训练数据的方法。这些改进进一步转化为在网页与操作系统代理下游任务中的显著性能提升。