Recent progress in GUI agents has substantially improved visual grounding, yet robust planning remains challenging, particularly when the environment deviates from a canonical initial state. In real applications, users often invoke assistance mid-workflow, where software may be partially configured, steps may have been executed in different orders, or the interface may differ from its default setup. Such task-state variability is pervasive but insufficiently evaluated in existing GUI benchmarks. To address this gap, we introduce WorldGUI, a benchmark covering ten widely used desktop and web applications with tasks instantiated under diverse, systematically constructed initial states. These variations capture realistic human-computer interaction settings and enable diagnostic evaluation of an agent's ability to recover, adapt plans, and handle non-default contexts. We further present WorldGUI-Agent, a simple and model-agnostic framework that organizes planning and execution around three critique stages, improving reliability in dynamic environments. Experiments demonstrate that state-of-the-art GUI agents exhibit substantial performance degradation under non-default initial conditions, revealing limited robustness and fragile planning behaviors. Our benchmark and framework provide a foundation for developing more adaptable and reliable GUI agents. The code and data are available at https://github.com/showlab/WorldGUI.
翻译:近期GUI智能体在视觉定位方面取得了显著进展,但稳健的规划仍具挑战性,尤其是在环境偏离标准初始状态时。在实际应用中,用户通常在工作流程中寻求帮助,此时软件可能已部分配置、步骤可能以不同顺序执行、或界面可能偏离默认设置。这种任务状态的多样性普遍存在,但在现有GUI基准测试中未得到充分评估。为填补这一空白,我们提出WorldGUI——一个覆盖十种常用桌面及网页应用程序的基准测试,其任务实例化于多样化、系统构建的初始状态下。这些变体捕捉了真实的人机交互场景,并能诊断性地评估智能体在非默认上下文中恢复、适应规划及处理问题的能力。我们进一步提出WorldGUI-Agent——一个简单且与模型无关的框架,通过围绕三个批判阶段组织规划与执行,提升动态环境下的可靠性。实验表明,最先进的GUI智能体在非默认初始条件下性能显著下降,暴露出鲁棒性有限与规划行为脆弱的问题。我们的基准测试与框架为开发更适应性强、更可靠的GUI智能体奠定了基础。代码与数据见https://github.com/showlab/WorldGUI。