Agents centered around Large Language Models (LLMs) are now capable of automating mobile device operations for users. After fine-tuning to learn a user's mobile operations, these agents can adhere to high-level user instructions online. They execute tasks such as goal decomposition, sequencing of sub-goals, and interactive environmental exploration, until the final objective is achieved. However, privacy concerns related to personalized user data arise during mobile operations, requiring user confirmation. Moreover, users' real-world operations are exploratory, with action data being complex and redundant, posing challenges for agent learning. To address these issues, in our practical application, we have designed interactive tasks between agents and humans to identify sensitive information and align with personalized user needs. Additionally, we integrated Standard Operating Procedure (SOP) information within the model's in-context learning to enhance the agent's comprehension of complex task execution. Our approach is evaluated on the new device control benchmark AitW, which encompasses 30K unique instructions across multi-step tasks, including application operation, web searching, and web shopping. Experimental results show that the SOP-based agent achieves state-of-the-art performance in LLMs without incurring additional inference costs, boasting an overall action success rate of 66.92\%. The code and data examples are available at https://github.com/alipay/mobile-agent.

翻译：围绕大语言模型（LLMs）构建的智能体现已能够为用户自动化移动设备操作。这些智能体在微调学习用户移动操作后，可在线遵循高级用户指令，执行目标分解、子目标排序以及交互式环境探索等任务，直至完成最终目标。然而，移动操作过程中涉及个性化用户数据的隐私问题需要用户确认。此外，用户的实际操作具有探索性，行动数据复杂且冗余，给智能体学习带来挑战。为解决上述问题，我们在实际应用中设计了智能体与用户之间的交互任务，用以识别敏感信息并匹配个性化用户需求。同时，我们通过模型上下文学习中融入标准操作流程（SOP）信息，增强智能体对复杂任务执行的理解能力。该方法在涵盖30,000条独特指令的多步骤任务新设备控制基准测试AitW（包括应用操作、网页搜索和网络购物）上进行了评估。实验结果表明，基于SOP的智能体在无需额外推理成本的情况下，在大语言模型中取得了最先进性能，整体行动成功率达66.92%。代码与数据示例已开源至https://github.com/alipay/mobile-agent。