GUI testing is significant in the SE community. Most existing frameworks are intrusive and only support some specific platforms. With the development of distinct scenarios, diverse embedded systems or customized operating systems on different devices do not support existing intrusive GUI testing frameworks. Some approaches adopt robotic arms to replace the interface invoking of mobile apps under test and use computer vision technologies to identify GUI elements. However, some challenges are unsolved. First, existing approaches assume that GUI screens are fixed so that they cannot be adapted to diverse systems with different screen conditions. Second, existing approaches use XY-plane robotic arms, which cannot flexibly simulate testing operations. Third, existing approaches ignore compatibility bugs and only focus on crash bugs. A more practical approach is required for the non-intrusive scenario. We propose a practical non-intrusive GUI testing framework with visual robotic arms. RoboTest integrates novel GUI screen and widget detection algorithms, adaptive to detecting screens of different sizes and then to extracting GUI widgets from the detected screens. Then, a set of testing operations is applied with a 4-DOF robotic arm, which effectively and flexibly simulates human testing operations. During app exploration, RoboTest integrates the Principle of Proximity-guided exploration strategy, choosing close widgets of the previous targets to reduce robotic arm movement overhead and improve exploration efficiency. RoboTest can effectively detect some compatibility bugs beyond crash bugs with a GUI comparison on different devices of the same test operations. We evaluate RoboTest with 20 mobile apps, with a case study on an embedded system. The results show that RoboTest can effectively, efficiently, and generally explore AUTs to find bugs and reduce exploration time overhead.

翻译：GUI测试在软件工程领域具有重要意义。现有框架多为侵入式，且仅支持特定平台。随着应用场景的多样化发展，不同设备上的嵌入式系统或定制操作系统无法支持现有的侵入式GUI测试框架。部分方法采用机械臂替代被测移动应用的界面调用，并利用计算机视觉技术识别GUI元素，但仍存在若干待解决挑战。首先，现有方法假设GUI屏幕固定，无法适应不同屏幕条件的多样化系统；其次，现有方法采用XY平面机械臂，无法灵活模拟测试操作；第三，现有方法仅关注崩溃缺陷而忽略兼容性缺陷。针对非侵入式场景，亟需更实用的解决方案。我们提出了一种基于视觉机械臂的实用非侵入式GUI测试框架RoboTest。该框架集成新型GUI屏幕与控件检测算法，可自适应检测不同尺寸的屏幕，并从中提取GUI控件。随后，通过四自由度机械臂执行一组测试操作，有效且灵活地模拟人工测试操作。在应用探索过程中，RoboTest集成邻近原则引导的探索策略，优先选择前序目标的邻近控件，以减少机械臂移动开销并提升探索效率。通过在不同设备上对相同测试操作进行GUI对比，RoboTest能够有效检测出超越崩溃缺陷的兼容性缺陷。我们使用20款移动应用对RoboTest进行评估，并在嵌入式系统上进行案例研究。结果表明，RoboTest能够有效、高效且通用地探索被测应用以发现缺陷，同时降低探索时间开销。