Screenshot-based mobile GUI agents can operate ordinary smartphone apps through the same visual interface as a human user, but this capability also turns every screen observation into a privacy boundary. During normal task execution, screenshots may expose contacts, messages, photos, files, recommendations, health cues, and other sensitive context that is unrelated to the user's request. We call this problem incidental visual privacy exposure. It is difficult to address with existing defenses: text anonymization misses many visual and inferential cues, while generic privacy masking can remove the evidence and controls that a GUI agent needs to complete the task. This paper presents CAPED, a context-aware pre-upload exposure control layer for mobile GUI agents. CAPED is designed as a phone-side protection layer: before screenshots are released to a remote multimodal agent, it extracts task requirements, uses screen context as a privacy prior, parses visible UI elements, and selectively exposes only content needed for the current task while masking incidental private content. We evaluate CAPED on AndroidWorld for broad task utility and with a controlled 28-task seeded privacy evaluation used as a measurement instrument for trajectory-level incidental leakage. In this seeded evaluation, Full CAPED reduces success-conditioned weighted seeded leakage from 0.766 under raw screenshots to 0.268 while preserving high task utility. A broader AndroidWorld run shows a remaining prototype-level utility cost, but the results show that task-driven selective exposure can reduce incidental visual leakage before screenshots are released to a remote GUI agent.
翻译:摘要:基于截图的移动GUI代理能够像人类用户一样,通过相同的视觉界面操作普通智能手机应用,但这种能力也使每一次屏幕观察都变成了隐私边界。在正常任务执行过程中,截图可能暴露与用户请求无关的联系人、消息、照片、文件、推荐内容、健康提示等敏感上下文。我们将此问题称为"偶然性视觉隐私暴露"。现有防御手段难以应对:文本匿名化会遗漏许多视觉和推断性线索,而通用隐私遮蔽则可能移除GUI代理完成任务所需的证据和控制要素。本文提出CAPED——一种面向移动GUI代理的上下文感知预上传暴露控制层。CAPED被设计为手机端保护层:在截图发送至远程多模态代理之前,它提取任务需求、以屏幕上下文作为隐私先验、解析可见UI元素,并选择性地仅暴露当前任务所需内容,同时遮蔽偶然性私人内容。我们在AndroidWorld上评估CAPED的任务通用性,并通过包含28个任务的受控种子隐私评估作为轨迹级偶然泄露的测量工具。在该种子评估中,完整版CAPED将成功条件加权种子泄露率从原始截图下的0.766降至0.268,同时保持了较高的任务效用。更广泛的AndroidWorld运行显示仍存在原型级效用成本,但结果表明,在截图发送至远程GUI代理之前,任务驱动的选择性暴露可以减少偶然性视觉泄露。