When translating UI design prototypes to code in industry, automatically generating code from design prototypes can expedite the development of applications and GUI iterations. However, in design prototypes without strict design specifications, UI components may be composed of fragmented elements. Grouping these fragmented elements can greatly improve the readability and maintainability of the generated code. Current methods employ a two-stage strategy that introduces hand-crafted rules to group fragmented elements. Unfortunately, the performance of these methods is not satisfying due to visually overlapped and tiny UI elements. In this study, we propose EGFE, a novel method for automatically End-to-end Grouping Fragmented Elements via UI sequence prediction. To facilitate the UI understanding, we innovatively construct a Transformer encoder to model the relationship between the UI elements with multi-modal representation learning. The evaluation on a dataset of 4606 UI prototypes collected from professional UI designers shows that our method outperforms the state-of-the-art baselines in the precision (by 29.75\%), recall (by 31.07\%), and F1-score (by 30.39\%) at edit distance threshold of 4. In addition, we conduct an empirical study to assess the improvement of the generated front-end code. The results demonstrate the effectiveness of our method on a real software engineering application. Our end-to-end fragmented elements grouping method creates opportunities for improving UI-related software engineering tasks.

翻译：在工业界将UI设计原型转化为代码的过程中，从设计原型自动生成代码可加速应用程序开发与图形用户界面迭代。然而，在缺乏严格设计规范的原型中，UI组件可能由碎片化元素构成。对这些碎片化元素进行分组能显著提升生成代码的可读性与可维护性。现有方法采用两阶段策略，通过人工规则对碎片化元素进行分组，但由于视觉重叠和微小UI元素的存在，其性能并不理想。本研究提出EGFE，一种通过UI序列预测自动实现碎片元素端到端分组的新方法。为促进UI理解，我们创新性地构建了一个Transformer编码器，通过多模态表示学习建模UI元素间的关系。在从专业UI设计师处收集的4606个UI原型数据集上的评估表明，在编辑距离阈值为4时，本方法在精确率（提升29.75%）、召回率（提升31.07%）和F1分数（提升30.39%）上均优于现有最优基线模型。此外，我们开展实证研究评估生成前端代码的改进效果，结果验证了本方法在实际软件工程应用中的有效性。所提出的端到端碎片元素分组方法为改进UI相关软件工程任务创造了新机遇。