Much of the previous work towards digital agents for graphical user interfaces (GUIs) has relied on text-based representations (derived from HTML or other structured data sources), which are not always readily available. These input representations have been often coupled with custom, task-specific action spaces. This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use -- via pixel-based screenshots and a generic action space corresponding to keyboard and mouse actions. Building upon recent progress in pixel-based pretraining, we show, for the first time, that it is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.

翻译：先前关于图形用户界面数字代理的大部分工作依赖于基于文本的表示（源自HTML或其他结构化数据源），而这些数据并非总是易于获取。这些输入表示通常与自定义的任务特定操作空间相结合。本文聚焦于创建能够使用人类普遍采用的相同概念界面——即基于像素的屏幕截图及对应于键盘和鼠标操作的通用操作空间——与数字世界交互的代理。基于像素级预训练的最新进展，我们首次证明，这类代理在基于GUI的指令遵循任务基准MiniWob++上的表现能够超越人类众包工作者。