Voice command interfaces (VCIs) have gained increasing importance, enabling hands-free and eyes-free interaction with digital devices. However, the inherent complexity in constructing effective voice interfaces has limited the VCIs' functionalities to only a small fraction of GUI applications and tasks. This paper presents AutoTask, a VCI capable of automating any task in any mobile application without configuration or modification from developers or end users. The primary challenge for AutoTask is the lack of knowledge, as it needs to accomplish unknown tasks (e.g., user commands) within an unknown environment (e.g., GUI). To address this challenge, AutoTask employs two strategies: (1) trial and error: AutoTask explores the GUI, attempts potential operation sequences, and recovers from errors through backtracking; (2) learning from the environment: AutoTask accumulates experiences during exploration and summarizes correct knowledge from these experiences. We implemented AutoTask on Android devices and conducted an evaluation study, which proved the feasibility of AutoTask.

翻译：摘要：语音命令界面（VCIs）的重要性日益凸显，它实现了对数字设备的免提和免视觉交互。然而，构建有效语音界面的固有复杂性将VCIs的功能限制在GUI应用程序和任务的极小部分。本文提出AutoTask，一种无需开发者或终端用户配置或修改即可自动化任何移动应用中任意任务的VCI。AutoTask面临的主要挑战是知识缺失——它需要在未知环境（如GUI）中完成未知任务（如用户指令）。为应对这一挑战，AutoTask采用两种策略：（1）试错：AutoTask探索GUI，尝试潜在的操作序列，并通过回溯从错误中恢复；（2）从环境学习：AutoTask在探索过程中积累经验，并从这些经验中归纳正确知识。我们在Android设备上实现了AutoTask并开展了评估研究，验证了其可行性。