This paper introduces DroidBot-GPT, a tool that utilizes GPT-like large language models (LLMs) to automate the interactions with Android mobile applications. Given a natural language description of a desired task, DroidBot-GPT can automatically generate and execute actions that navigate the app to complete the task. It works by translating the app GUI state information and the available actions on the smartphone screen to natural language prompts and asking the LLM to make a choice of actions. Since the LLM is typically trained on a large amount of data including the how-to manuals of diverse software applications, it has the ability to make reasonable choices of actions based on the provided information. We evaluate DroidBot-GPT with a self-created dataset that contains 33 tasks collected from 17 Android applications spanning 10 categories. It can successfully complete 39.39% of the tasks, and the average partial completion progress is about 66.76%. Given the fact that our method is fully unsupervised (no modification required from both the app and the LLM), we believe there is great potential to enhance automation performance with better app development paradigms and/or custom model training.

翻译：本文介绍了DroidBot-GPT，一种利用GPT类大语言模型自动与安卓移动应用进行交互的工具。给定目标任务的自然语言描述，DroidBot-GPT可自动生成并执行操作，通过导航应用界面完成任务。其工作原理是将应用图形用户界面状态信息及智能手机屏幕上的可用操作转化为自然语言提示，并请求大语言模型做出操作选择。由于大语言模型通常经过海量数据训练，包含各类软件应用的使用手册，因此具备依据所提供信息做出合理操作决策的能力。我们通过自建数据集对DroidBot-GPT进行评估，该数据集包含来自17个安卓应用（覆盖10个类别）的33项任务。实验结果显示，该工具能成功完成39.39%的任务，平均部分完成进度约为66.76%。鉴于本方法完全无监督（无需对应用和大语言模型进行任何修改），我们相信通过改进应用开发范式或定制模型训练，自动化性能具有极大的提升潜力。