As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on: https://github.com/meituan-longcat/SOP-Maze.

翻译：随着大语言模型（LLMs）作为领域特定智能体被广泛部署，已有许多基准测试被提出，以评估其在现实场景中遵循指令和做出决策的能力。然而，业务场景通常涉及复杂的标准操作程序（SOPs），而在此类背景下对大语言模型能力的评估尚未得到充分探索。为弥补这一空白，我们提出了SOP-Maze，这是一个基于真实业务数据构建的基准测试，它被改编为来自23个复杂SOP场景的397个实例和3422个子任务。我们进一步将SOP任务划分为两大类：横向根系（LRS），代表需要精确选择的宽选项任务；以及核心根系（HRS），其强调具有复杂分支的深度逻辑推理。大量实验表明，几乎所有最先进的模型在SOP-Maze上都表现不佳。我们进行了全面分析，并识别出三个关键错误类别：（i）路径盲区：难以遵循流程；（ii）对话脆弱性：无法处理真实对话的细微差别；以及（iii）计算错误：在复杂情境下进行时间或算术推理时出错。这项系统研究探索了大语言模型在挑战广度和深度的SOP任务上的表现，为提升模型能力提供了新的见解。我们的工作已在以下地址开源：https://github.com/meituan-longcat/SOP-Maze。