Navigating mobile User Interface (UI) applications using large language and vision models based on high-level goal instructions is emerging as an important research field with significant practical implications, such as digital assistants and automated UI testing. To evaluate the effectiveness of existing models in mobile UI navigation, benchmarks are required and widely used in the literature. Although multiple benchmarks have been recently established for evaluating functional correctness being judged as pass or fail, they fail to address the need for multi-dimensional evaluation of the entire UI navigation process. Furthermore, other exiting related datasets lack an automated and robust benchmarking suite, making the evaluation process labor-intensive and error-prone. To address these issues, in this paper, we propose a new benchmark named Sphinx for multi-dimensional evaluation of existing models in practical UI navigation. Sphinx provides a fully automated benchmarking suite that enables reproducibility across real-world mobile apps and employs reliable evaluators to assess model progress. In addition to functional correctness, Sphinx includes comprehensive toolkits for multi-dimensional evaluation, such as invariant-based verification, knowledge probing, and knowledge-augmented generation to evaluate model capabilities including goal understanding, knowledge and planning, grounding, and instruction following, ensuring a thorough assessment of each sub-process in mobile UI navigation. We benchmark 8 large language and multi-modal models with 13 different configurations on Sphinx. Evaluation results show that all these models struggle on Sphinx, and fail on all test generation tasks. Our further analysis of the multi-dimensional evaluation results underscores the current progress and highlights future research directions to improve a model's effectiveness for mobile UI navigation.

翻译：基于高级目标指令、利用大型语言与视觉模型进行移动用户界面（UI）应用程序导航，正成为一个重要的研究领域，具有显著的实际应用价值，例如数字助手和自动化UI测试。为了评估现有模型在移动UI导航中的有效性，基准测试在文献中被广泛需求和使用。尽管近期已建立多个用于评估功能正确性（以通过或失败为判断标准）的基准，但它们未能满足对整个UI导航过程进行多维度评估的需求。此外，其他现有相关数据集缺乏自动化且稳健的基准测试套件，导致评估过程劳动密集且容易出错。为解决这些问题，本文提出一个名为Sphinx的新基准，用于对现有模型在实际UI导航中进行多维度评估。Sphinx提供了一个完全自动化的基准测试套件，能够在真实世界移动应用中实现可复现性，并采用可靠的评估器来评估模型进展。除了功能正确性之外，Sphinx还包含用于多维度评估的综合工具包，例如基于不变量的验证、知识探测和知识增强生成，以评估模型在目标理解、知识与规划、接地以及指令遵循等方面的能力，确保对移动UI导航中每个子过程进行全面评估。我们在Sphinx上对8个大型语言和多模态模型的13种不同配置进行了基准测试。评估结果表明，所有这些模型在Sphinx上都表现不佳，并且在所有测试生成任务中均告失败。我们对多维度评估结果的进一步分析，揭示了当前进展并指出了未来研究方向，以提升模型在移动UI导航中的有效性。