Automated software testing is integral to the software development process, streamlining workflows and ensuring product reliability. Visual testing within this context, especially concerning user interface (UI) and user experience (UX) validation, stands as one of crucial determinants of overall software quality. Nevertheless, conventional methods like pixel-wise comparison and region-based visual change detection fall short in capturing contextual similarities, nuanced alterations, and understanding the spatial relationships between UI elements. In this paper, we introduce a novel graph-based method for visual change detection in software test automation. Leveraging a machine learning model, our method accurately identifies UI controls from software screenshots and constructs a graph representing contextual and spatial relationships between the controls. This information is then used to find correspondence between UI controls within screenshots of different versions of a software. The resulting graph encapsulates the intricate layout of the UI and underlying contextual relations, providing a holistic and context-aware model. This model is finally used to detect and highlight visual regressions in the UI. Comprehensive experiments on different datasets showed that our change detector can accurately detect visual software changes in various simple and complex test scenarios. Moreover, it outperformed pixel-wise comparison and region-based baselines by a large margin in more complex testing scenarios. This work not only contributes to the advancement of visual change detection but also holds practical implications, offering a robust solution for real-world software test automation challenges, enhancing reliability, and ensuring the seamless evolution of software interfaces.

翻译：自动化软件测试是软件开发流程的核心组成部分，能够优化工作流程并确保产品可靠性。在此背景下，视觉测试（尤其是用户界面和用户体验验证）是决定整体软件质量的关键因素之一。然而，传统的逐像素对比和基于区域的视觉变化检测方法，在捕捉上下文相似性、细微变化以及理解UI元素之间的空间关系方面存在不足。本文提出了一种新颖的基于图的方法，用于软件测试自动化中的视觉变化检测。该方法利用机器学习模型，从软件截图中精确识别UI控件，并构建一个表示控件间上下文与空间关系的图结构。这些信息随后被用于寻找不同软件版本截图中UI控件之间的对应关系。生成的图结构封装了UI的复杂布局及其潜在的上下文关联，提供了一个全面的、具备情境感知能力的模型。最终，该模型被用于检测并高亮显示UI中的视觉回归问题。在多个数据集上的综合实验表明，我们的变化检测器能够在多种简单和复杂的测试场景中准确检测软件视觉变化。此外，在更复杂的测试场景中，其性能显著优于逐像素对比和基于区域的基线方法。这项工作不仅推动了视觉变化检测技术的发展，还具有实际应用价值：为真实场景中的软件测试自动化挑战提供了稳健的解决方案，增强了软件可靠性，并确保了软件界面的无缝演进。