There is a growing demand for mobile user interface (UI) automation, driven by its broad applications across industries. With the advent of visual language models (VLMs), GUI automation has progressed from generating text-based instructions for humans to autonomously executing tasks, thus optimizing automation workflows. Recent approaches leverage VLMs for this problem due to their ability to 1) process on-screen content directly, 2) remain independent of device-specific APIs by utilizing human actions (e.g., clicks, typing), and 3) apply real-world contextual knowledge for task understanding. However, these models often have trouble accurately identifying widgets and determining actions due to limited spatial information in vision encoder features. Additionally, top-performing models are often large, requiring extensive training and resulting in inference delays. In this work, we introduce AFRAgent, an instruct-BLIP-based multimodal architecture that achieves superior performance in GUI automation while being less than one-fourth the size of its nearest competitor. To enhance image embeddings in the large language model (LLM) pipeline, we propose an adaptive feature renormalization-based (a token-level affine transformation) technique that effectively enriches low-resolution image embeddings and fuses high-resolution details. We evaluate AFRAgent on Meta-GUI and AITW benchmarks, establishing a new state-of-the-art baseline for smartphone automation.

翻译：移动用户界面（UI）自动化的需求日益增长，因其在各行业的广泛应用而备受关注。随着视觉语言模型（VLMs）的出现，GUI自动化已从生成基于文本的人工指令发展为自主执行任务，从而优化了自动化工作流程。近期方法利用VLMs解决此问题，得益于其能够：1）直接处理屏幕内容；2）通过模拟人类操作（如点击、输入）保持与设备特定API的独立性；3）运用现实世界情境知识进行任务理解。然而，由于视觉编码器特征中空间信息有限，这些模型常难以准确定位界面组件并确定操作。此外，性能最优的模型通常规模庞大，需要大量训练并导致推理延迟。本研究提出AFRAgent，一种基于instruct-BLIP的多模态架构，其在GUI自动化中实现卓越性能，而模型规模仅为最接近竞争者的四分之一以下。为增强大语言模型（LLM）流程中的图像嵌入，我们提出一种基于自适应特征重归一化（即令牌级仿射变换）的技术，有效增强低分辨率图像嵌入并融合高分辨率细节。我们在Meta-GUI和AITW基准测试中评估AFRAgent，为智能手机自动化建立了新的最先进基准。