Automatically constructing GUI groups of different granularities constitutes a critical intelligent step towards automating GUI design and implementation tasks. Specifically, in the industrial GUI-to-code process, fragmented layers may decrease the readability and maintainability of generated code, which can be alleviated by grouping semantically consistent fragmented layers in the design prototypes. This study aims to propose a graph-learning-based approach to tackle the fragmented layer grouping problem according to multi-modal information in design prototypes. Our graph learning module consists of self-attention and graph neural network modules. By taking the multimodal fused representation of GUI layers as input, we innovatively group fragmented layers by classifying GUI layers and regressing the bounding boxes of the corresponding GUI components simultaneously. Experiments on two real-world datasets demonstrate that our model achieves state-of-the-art performance. A further user study is also conducted to validate that our approach can assist an intelligent downstream tool in generating more maintainable and readable front-end code.

翻译：自动构建不同粒度的GUI分组是实现GUI设计与实现任务自动化的关键智能步骤。具体而言，在工业级GUI到代码的转换流程中，碎片化的图层会降低生成代码的可读性与可维护性，而通过对设计原型中语义一致的碎片化图层进行分组可缓解此问题。本研究旨在提出一种基于图学习的方法，依据设计原型中的多模态信息解决碎片化图层分组问题。我们的图学习模块由自注意力机制与图神经网络模块构成。通过将GUI图层的多模态融合表征作为输入，我们创新性地通过同时分类GUI图层并回归对应GUI组件的边界框来实现碎片化图层分组。在两个真实数据集上的实验表明，我们的模型达到了最先进的性能。进一步的用户研究也证实，我们的方法能够辅助智能下游工具生成更具可维护性与可读性的前端代码。