Recent studies have demonstrated the success of foundation agents in specific tasks or scenarios. However, existing agents cannot generalize across different scenarios, mainly due to their diverse observation and action spaces and semantic gaps, or reliance on task-specific resources. In this work, we propose the General Computer Control (GCC) setting: building foundation agents that can master any computer task by taking only screen images (and possibly audio) of the computer as input, and producing keyboard and mouse operations as output, similar to human-computer interaction. To target GCC, we propose Cradle, an agent framework with strong reasoning abilities, including self-reflection, task inference, and skill curation, to ensure generalizability and self-improvement across various tasks. To demonstrate the capabilities of Cradle, we deploy it in the complex AAA game Red Dead Redemption II, serving as a preliminary attempt towards GCC with a challenging target. Our agent can follow the main storyline and finish real missions in this complex AAA game, with minimal reliance on prior knowledge and application-specific resources. The project website is at https://baai-agents.github.io/Cradle/.

翻译：近期研究已证明基础智能体在特定任务或场景中的成功。然而，现有智能体无法跨场景泛化，主要受限于其多样化的观测空间与动作空间、语义鸿沟，或对任务特定资源的依赖。本研究提出通用计算机控制（GCC）设定：构建能够通过仅输入计算机屏幕图像（及可能音频）作为观测，输出键盘与鼠标操作作为交互（类似于人机交互）的基础智能体，从而掌握任意计算机任务。针对GCC目标，我们提出Cradle智能体框架，该框架具备强推理能力，包括自我反思、任务推演与技能编排，以保障跨任务泛化性与自我改进能力。为验证Cradle的能力，我们在复杂3A游戏《荒野大镖客：救赎2》中部署该框架，作为面向GCC的初步尝试与挑战性目标。该智能体能在极少量依赖先验知识与特定应用资源的情况下，遵循主线剧情并完成该复杂3A游戏中的真实任务。项目网站：https://baai-agents.github.io/Cradle/。