We present a domain-grounded framework and benchmark for tool-aware plan generation in contact centers, where answering a query for business insights, our target use case, requires decomposing it into executable steps over structured tools (Text2SQL (T2S)/Snowflake) and unstructured tools (RAG/transcripts) with explicit depends_on for parallelism. Our contributions are threefold: (i) a reference-based plan evaluation framework operating in two modes - a metric-wise evaluator spanning seven dimensions (e.g., tool-prompt alignment, query adherence) and a one-shot evaluator; (ii) a data curation methodology that iteratively refines plans via an evaluator->optimizer loop to produce high-quality plan lineages (ordered plan revisions) while reducing manual effort; and (iii) a large-scale study of 14 LLMs across sizes and families for their ability to decompose queries into step-by-step, executable, and tool-assigned plans, evaluated under prompts with and without lineage. Empirically, LLMs struggle on compound queries and on plans exceeding 4 steps (typically 5-15); the best total metric score reaches 84.8% (Claude-3-7-Sonnet), while the strongest one-shot match rate at the "A+" tier (Extremely Good, Very Good) is only 49.75% (o3-mini). Plan lineage yields mixed gains overall but benefits several top models and improves step executability for many. Our results highlight persistent gaps in tool-understanding, especially in tool-prompt alignment and tool-usage completeness, and show that shorter, simpler plans are markedly easier. The framework and findings provide a reproducible path for assessing and improving agentic planning with tools for answering data-analysis queries in contact-center settings.

翻译：本文提出了一个领域扎根的框架与基准，用于联络中心场景下的工具感知规划生成。在该场景中，回答业务洞察查询（我们的目标用例）需要将其分解为在结构化工具（Text2SQL (T2S)/Snowflake）和非结构化工具（RAG/transcripts）上可执行的步骤，并明确指定依赖关系（depends_on）以实现并行化。我们的贡献包括三个方面：(i) 一个基于参考的规划评估框架，其包含两种模式——涵盖七个维度（例如，工具提示对齐、查询遵循度）的度量评估器和一个一次性评估器；(ii) 一种数据整理方法，通过“评估器->优化器”循环迭代优化规划，以生成高质量的规划谱系（有序的规划修订版本），同时减少人工工作量；(iii) 一项大规模研究，评估了14个不同规模和系列的LLM将查询分解为逐步、可执行且分配了工具的规划的能力，评估时使用了包含谱系信息和不包含谱系信息的提示。实验结果表明，LLM在处理复合查询以及步骤数超过4步（通常为5-15步）的规划时存在困难；最佳总度量得分达到84.8%（Claude-3-7-Sonnet），而在“A+”级别（极好、很好）的最强一次性匹配率仅为49.75%（o3-mini）。规划谱系总体上带来了混合的收益，但对几个顶级模型有益，并改善了许多规划步骤的可执行性。我们的结果凸显了在工具理解方面存在的持续差距，特别是在工具提示对齐和工具使用完整性方面，并表明更短、更简单的规划明显更容易处理。该框架与发现为评估和改进联络中心场景下用于回答数据分析查询的具身工具规划提供了一条可复现的路径。