Online reinforcement learning (RL) serves as an effective method for enhancing the capabilities of Android agents. However, guiding agents to learn through online interaction is prohibitively expensive due to the high latency of emulators and the sample inefficiency of existing RL algorithms. We identify a fundamental limitation in current approaches: the Single State Single Action paradigm, which updates the policy with one-to-one state-action pairs from online one-way rollouts without fully exploring each costly emulator state. In this paper, we propose Android Coach, a novel framework that shifts the training paradigm to Single State Multiple Actions, allowing the agent to sample and utilize multiple actions for a single online state. We enable this without additional emulator overhead by learning a critic that estimates action values. To ensure the critic serves as a reliable coach, we integrate a process reward model and introduce a group-wise advantage estimator based on the averaged critic outputs. Extensive experiments demonstrate the effectiveness and efficiency of Android Coach: it achieves 7.5% and 8.3% success rate improvements on AndroidLab and AndroidWorld over UI-TARS-1.5-7B, and attains 1.4x higher training efficiency than Single State Single Action methods PPO and GRPO at matched success rates.

翻译：在线强化学习是提升安卓智能体能力的有效方法。然而，由于模拟器的高时延和现有强化学习算法的样本效率低下，通过在线交互引导智能体学习的成本极高。我们识别出现有方法的一个根本性局限：单状态单动作范式，该范式仅从单向在线交互中获取一对一状态-动作对来更新策略，未能充分探索每个代价高昂的模拟器状态。本文提出Android Coach这一新型框架，将训练范式转变为单状态多动作，使智能体能够为单个在线状态采样并利用多个动作。我们通过学习一个评估动作价值的评判器，在不增加模拟器开销的情况下实现这一目标。为确保评判器作为可靠的指导者，我们集成了过程奖励模型，并引入基于评判器平均输出的分组优势估计器。大量实验证明了Android Coach的有效性和高效性：在AndroidLab和AndroidWorld上，相比UI-TARS-1.5-7B，其成功率分别提升7.5%和8.3%；在匹配成功率下，其训练效率比单状态单动作方法PPO和GRPO高出1.4倍。