Automatically generating UI code from webpage design visions can significantly alleviate the burden of developers, enabling beginner developers or designers to directly generate Web pages from design diagrams. Currently, prior research has accomplished the objective of generating UI code from rudimentary design visions or sketches through designing deep neural networks. Inspired by the groundbreaking advancements achieved by Multimodal Large Language Models (MLLMs), the automatic generation of UI code from high-fidelity design images is now emerging as a viable possibility. Nevertheless, our investigation reveals that existing MLLMs are hampered by the scarcity of authentic, high-quality, and large-scale datasets, leading to unsatisfactory performance in automated UI code generation. To mitigate this gap, we present a novel dataset, termed VISION2UI, extracted from real-world scenarios, augmented with comprehensive layout information, tailored specifically for finetuning MLLMs in UI code generation. Specifically, this dataset is derived through a series of operations, encompassing collecting, cleaning, and filtering of the open-source Common Crawl dataset. In order to uphold its quality, a neural scorer trained on labeled samples is utilized to refine the data, retaining higher-quality instances. Ultimately, this process yields a dataset comprising 2,000 (Much more is coming soon) parallel samples encompassing design visions and UI code. The dataset is available at https://huggingface.co/datasets/xcodemind/vision2ui.

翻译：从网页设计图自动生成UI代码能显著减轻开发人员负担，使初级开发者或设计师能够直接从设计图生成网页。当前，已有研究通过设计深度神经网络实现了从基础设计图或草稿生成UI代码的目标。受多模态大语言模型突破性进展的启发，从高保真设计图自动生成UI代码正成为可行的技术方向。然而，我们的研究发现，现有MLLMs受限于缺乏真实、高质量且大规模的数据集，在自动化UI代码生成任务中表现不佳。为弥补这一差距，我们提出了一个名为VISION2UI的新型数据集，该数据集从真实场景中提取，并补充了全面的布局信息，专门用于微调MLLMs以生成UI代码。具体而言，该数据集通过对开源Common Crawl数据集进行采集、清洗和筛选等一系列操作生成。为确保数据质量，我们利用基于标注样本训练的神经评分器对数据进行精炼，保留高质量样本。最终，该数据集包含2000个（更多样本即将发布）涵盖设计图与UI代码的并行样本。数据集地址为https://huggingface.co/datasets/xcodemind/vision2ui。