At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write prompts for each step, and repeatedly revise the logic as requirements evolve-making development costly, time-consuming, and error-prone. To study whether large language models can automate this multi-round interaction process, we introduce Chat2Workflow, a benchmark for generating executable visual workflows directly from natural language, and propose a robust agentic framework to mitigate recurrent execution errors. Chat2Workflow is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially under complex or changing requirements. Although our agentic framework yields up to 5.34% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.

翻译：当前，可执行的可视化工作流已成为实际工业部署中的主流范式，具备强大的可靠性和可控性。然而，在实践中，此类工作流几乎完全依赖人工构建：开发者需精心设计工作流、为每个步骤编写提示词，并随需求变化反复修订逻辑——导致开发成本高昂、耗时且易出错。为探究大语言模型能否自动化这一多轮交互过程，我们提出Chat2Workflow——一个从自然语言直接生成可执行可视化工作流的基准，并设计了一套稳健的智能体框架以缓解反复出现的执行错误。Chat2Workflow基于大量真实商业工作流构建，每个实例所生成的工作流均可转换并直接部署至Dify、Coze等实际工作流平台。实验结果表明，尽管当前最优语言模型通常能理解高层意图，但在生成正确、稳定且可执行的工作流方面仍存在不足，尤其在复杂或动态需求场景下。尽管我们的智能体框架带来了最高5.34%的解决率提升，但剩余的真实场景差距使Chat2Workflow成为推动工业级自动化的基础。代码已开源：https://github.com/zjunlp/Chat2Workflow。