Texts, widgets, and images on a UI page do not work separately. Instead, they are partitioned into groups to achieve certain interaction functions or visual information. Existing studies on UI elements grouping mainly focus on a specific single UI-related software engineering task, and their groups vary in appearance and function. In this case, we propose our semantic component groups that pack adjacent text and non-text elements with similar semantics. In contrast to those task-oriented grouping methods, our semantic component group can be adopted for multiple UI-related software tasks, such as retrieving UI perceptual groups, improving code structure for automatic UI-to-code generation, and generating accessibility data for screen readers. To recognize semantic component groups on a UI page, we propose a robust, deep learning-based vision detector, UISCGD, which extends the SOTA deformable-DETR by incorporating UI element color representation and a learned prior on group distribution. The model is trained on our UI screenshots dataset of 1988 mobile GUIs from more than 200 apps in both iOS and Android platforms. The evaluation shows that our UISCGD achieves 6.1\% better than the best baseline algorithm and 5.4 \% better than deformable-DETR in which it is based.

翻译：文本、控件和图像在UI页面上并非独立工作，而是被分组以实现特定的交互功能或视觉信息。现有关于UI元素分组的研究主要关注特定的单一UI相关软件工程任务，且其组在形态和功能上各异。基于此，我们提出语义组件组，将具有相似语义的相邻文本和非文本元素打包在一起。与那些面向任务的分组方法不同，我们的语义组件组可适用于多种UI相关软件任务，例如检索UI感知组、改进自动UI到代码生成的代码结构，以及为屏幕阅读器生成无障碍数据。为了识别UI页面上的语义组件组，我们提出了一种鲁棒的基于深度学习的视觉检测器UISCGD，该检测器通过引入UI元素颜色表示和组分布的先验学习来扩展最先进的deformable-DETR。该模型在我们从超过200个iOS和Android平台应用收集的1988个移动GUI的UI截图数据集上进行训练。评估结果表明，我们的UISCGD比最佳基线算法性能提升6.1%，比其所基于的deformable-DETR提升5.4%。