Workflows are a fundamental component of automation in enterprise platforms, enabling the orchestration of tasks, data processing, and system integrations. Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. To simplify this process, we explore the use of generative foundation models, particularly vision-language models (VLMs), to automatically generate structured workflows from visual inputs. Translating hand-drawn sketches or computer-generated diagrams into executable workflows is challenging due to the ambiguity of free-form drawings, variations in diagram styles, and the difficulty of inferring execution logic from visual elements. To address this, we introduce StarFlow, a framework for generating structured workflow outputs from sketches using vision-language models. We curate a diverse dataset of workflow diagrams -- including synthetic, manually annotated, and real-world samples -- to enable robust training and evaluation. We finetune and benchmark multiple vision-language models, conducting a series of ablation studies to analyze the strengths and limitations of our approach. Our results show that finetuning significantly enhances structured workflow generation, outperforming large vision-language models on this task.

翻译：工作流是企业平台自动化的基本组成部分，能够实现任务编排、数据处理和系统集成。尽管工作流被广泛使用，但其构建过程可能较为复杂，通常需要通过低代码平台或可视化编程工具进行手动配置。为简化这一过程，我们探索利用生成式基础模型，特别是视觉-语言模型（VLMs），从视觉输入自动生成结构化工作流。将手绘草图或计算机生成的图表转换为可执行工作流具有挑战性，原因在于自由形式绘图的模糊性、图表风格的多样性以及从视觉元素推断执行逻辑的困难。为此，我们提出了StarFlow——一个利用视觉-语言模型从草图生成结构化工作流输出的框架。我们构建了一个多样化的工作流图表数据集（包括合成样本、人工标注样本和真实世界样本），以支持稳健的训练与评估。我们对多个视觉-语言模型进行了微调与基准测试，并通过一系列消融实验分析了该方法的优势与局限性。实验结果表明，微调显著提升了结构化工作流的生成质量，在此任务上优于大型视觉-语言模型。